Data query method, device and electronic equipment

Abstract

The present invention discloses a data query method, device and electronic device. The method comprises: obtaining a query word, wherein the query word is used to determine the data to be queried in the current data set; based on the query word, performing a data query in a sampling pool corresponding to the current data set to obtain a sampling query result, wherein the sampling query result includes at least one target sampling document in the sampling pool that matches the query word; using the sampling query result, determining the subject data segment to be queried from the clustering result corresponding to the current data set, wherein the clustering result includes multiple data fragments, and each data fragment of the multiple data fragments contains at least one subject data segment; based on the query word, performing a data query in the subject data segment to be queried to obtain a target query result. The present invention solves the technical problems of the data query method provided by the related art, such as large resource consumption, large result error and slow query speed.

Description

Technical Field

[0001] The present invention relates to the field of big data, and in particular to a data query method, device and electronic equipment. Background Art

[0002] Search engines are important application tools of big data technology, which mainly retrieve relevant data by calculating the similarity score between queries and documents. Since users usually only care about whether the documents within a certain ranking in the search result set meet their expectations, and users have high requirements for the search speed and the error of the search results, it is very important to reduce the query response time of the search engine, improve the system throughput and the relevance of the search results to improve the user experience.

[0003] In the prior art, Elastic-Search (ES for short) is an open source search engine based on Lucene (an open source full-text search engine toolkit), which performs operations on massive data in a distributed manner through REST API (also known as RESTful API, an application programming interface that follows the Representational State Transfer architecture specification). However, the search engine system has the following problems:

[0004] (1) Since each index consists of one or more shards, which can be distributed on different nodes, there are problems of high network bandwidth consumption and high system load overhead when querying. In addition, when the server pressure increases sharply, it may cause query timeout and poor system stability.

[0005] (2) Since the entire data set of the search engine is divided into shards in the form of indexes and distributed to various data nodes, when querying, all shards of the index are often queried and the returned results of each shard are summarized and rearranged. In addition, the data distribution is random. Therefore, when the expected optimal query result happens to be distributed on a shard that has not been queried, it will cause a large error in the search results.

[0006] (3) Since ES will update the shards regularly, when updating the shards, it is necessary to search across a large number of shards, which will result in low system throughput, slow search speed, and unstable cluster.

[0007] In order to solve the above problems, the existing technology usually adopts methods such as shutting down services, limiting current, and downgrading policies. Obviously, the methods provided by the existing technology greatly reduce the user experience.

[0008] As can be seen from the above, there is currently no effective solution to the problems of large resource consumption, large result errors, and slow search speed in the above data query method. Summary of the invention

[0009] The embodiments of the present invention provide a data query method, device and electronic device to at least solve the technical problems of the data query method provided by the related art, such as large resource consumption, large result error and slow query speed.

[0010] According to one aspect of an embodiment of the present invention, there is provided a data query method, comprising:

[0011] Obtain query words, wherein the query words are used to determine the data to be queried in the current data set; based on the query words, perform data query in the sampling pool corresponding to the current data set to obtain sampling query results, wherein the sampling query results include at least one target sampling document in the sampling pool that matches the query words; using the sampling query results, determine the subject data segment to be queried from the clustering results corresponding to the current data set, wherein the clustering results include multiple data fragments, each of the multiple data fragments contains at least one subject data segment; based on the query words, perform data query in the subject data segment to be queried to obtain the target query result.

[0012] Optionally, the current data set is randomly sampled to obtain a sampling pool, wherein the sampling pool includes a plurality of candidate sampling documents; semantic clustering is performed on the plurality of documents in the current data set to obtain a clustering result.

[0013] Optionally, based on the query terms, data query is performed in the sampling pool corresponding to the current data set to obtain the sampling query results, including: calculating the matching degree between the query terms and multiple candidate sampling documents in the sampling pool to obtain a matching order; based on the matching order, selecting at least one target sampling document in the sampling query results from the candidate sampling documents.

[0014] Optionally, semantic clustering is performed on multiple documents in the current data set to obtain clustering results, including: using a document semantic model to extract features from each of the multiple documents in the current data set to obtain a feature vector for each of the multiple documents, wherein the document semantic model is a neural network model obtained by offline training based on the current data set using an attention mechanism; based on the feature vector, semantic clustering is performed on the current data set to obtain a clustering result.

[0015] Optionally, based on the feature vector, semantic clustering is performed on the current data set to obtain a clustering result including: based on the feature vector of each document in the multiple documents, a preset clustering algorithm is used to semantically cluster the multiple documents in the current data set to obtain multiple topic data segments, wherein the topic centroid of each topic data segment in the multiple topic data segments corresponds to a semantic clustering center; and the multiple topic data segments are fragmented and combined to obtain multiple data fragments in the clustering result.

[0016] Optionally, a document semantic model is used to perform feature extraction on the newly added documents in the current data set to extract feature vectors of the newly added documents; based on the feature distance between the feature vectors of the newly added documents and the topic centroid of each topic data segment in the multiple topic data segments, a topic data segment to be added is determined from the multiple topic data segments; and the newly added documents are added to the topic data segment to be added.

[0017] Optionally, adding a new document to the added subject data segment includes: writing the new document and corresponding subordinate information into the buffer of the current data set, wherein the subordinate information is used to describe the subject data segment to be added corresponding to the new document and the data shard to be added to which the subject data segment to be added belongs; based on the subordinate information, merging the new document into the subject data segment to be added in the data shard to be added.

[0018] Optionally, utilizing the sampling query results, determining the subject data segment to be queried from the clustering results corresponding to the current data set includes: determining the candidate subject data segment to which each target sampling document in the sampling query results belongs from the clustering results; performing weighted voting on the candidate subject data segments based on the sampling query results and the matching order to obtain a voting result; in response to the voting result satisfying a preset condition, determining the subject data segment to be queried according to the voting result.

[0019] According to another aspect of an embodiment of the present invention, there is further provided a data query device, comprising:

[0020] An acquisition module is used to acquire query words, wherein the query words are used to determine the data to be queried in the current data set; a first query module is used to perform data query in a sampling pool corresponding to the current data set based on the query words to obtain a sampling query result, wherein the sampling query result includes at least one target sampling document in the sampling pool that matches the query words; a determination module is used to use the sampling query result to determine the subject data segment to be queried from the clustering result corresponding to the current data set, wherein the clustering result includes multiple data fragments, and each data fragment of the multiple data fragments contains at least one subject data segment; a second query module is used to perform data query in the subject data segment to be queried based on the query words to obtain a target query result.

[0021] Optionally, the above data query method also includes: a clustering module, which is used to randomly sample the current data set to obtain a sampling pool, wherein the sampling pool includes multiple candidate sampling documents; and semantically cluster the multiple documents in the current data set to obtain a clustering result.

[0022] Optionally, the above-mentioned first query module is also used to: based on the query terms, perform data query in the sampling pool corresponding to the current data set, and obtain the sampling query results including: calculating the matching degree between the query terms and multiple candidate sampling documents in the sampling pool to obtain a matching order; based on the matching order, select at least one target sampling document in the sampling query results from the candidate sampling documents.

[0023] Optionally, the above-mentioned clustering module is also used to: perform semantic clustering on multiple documents in the current data set, and obtain the clustering results including: using the document semantic model to extract features for each of the multiple documents in the current data set to obtain a feature vector for each of the multiple documents, wherein the document semantic model is a neural network model obtained by offline training based on the current data set using an attention mechanism; based on the feature vector, perform semantic clustering on the current data set to obtain a clustering result.

[0024] Optionally, the above-mentioned clustering module is also used to: perform semantic clustering on the current data set based on the feature vector, and obtain the clustering results including: based on the feature vector of each document in the multiple documents, use a preset clustering algorithm to semantically cluster the multiple documents in the current data set to obtain multiple topic data segments, wherein the topic centroid of each topic data segment in the multiple topic data segments corresponds to a semantic clustering center; and perform fragmentation and combination on the multiple topic data segments to obtain multiple data fragments in the clustering results.

[0025] Optionally, the above-mentioned clustering module is also used to: use the document semantic model to perform feature extraction on the new documents of the current data set, and extract the feature vector of the new documents; determine the topic data segment to be added from multiple topic data segments based on the feature distance between the feature vector of the new document and the topic centroid of each topic data segment in multiple topic data segments; add the new document to the topic data segment to be added.

[0026] Optionally, the above-mentioned clustering module is also used to: add the newly added document to the added topic data segment, including: writing the newly added document and the corresponding subordinate information into the buffer of the current data set, wherein the subordinate information is used to describe the to-be-added topic data segment corresponding to the newly added document and the to-be-added data shard to which the to-be-added topic data segment belongs; according to the subordinate information, merge the newly added document into the to-be-added topic data segment in the to-be-added data shard.

[0027] Optionally, the above-mentioned determination module is also used to: use the sampling query results to determine the subject data segment to be queried from the clustering results corresponding to the current data set, including: determining the candidate subject data segment to which each target sampling document in the sampling query results belongs from the clustering results; based on the sampling query results and the matching order, performing weighted voting on the candidate subject data segments to obtain the voting results; in response to the voting results satisfying preset conditions, determining the subject data segment to be queried according to the voting results.

[0028] According to another aspect of an embodiment of the present invention, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the aforementioned data query methods.

[0029] In an embodiment of the present invention, a query word is first obtained, wherein the query word is used to determine the data to be queried in the current data set. Then, based on the query word, data query is performed in the sampling pool corresponding to the current data set to obtain a sampling query result, wherein the sampling query result includes at least one target sampling document in the sampling pool that matches the query word. Then, using the sampling query result, the subject data segment to be queried is determined from the clustering result corresponding to the current data set, wherein the clustering result includes multiple data fragments, and each data fragment of the multiple data fragments contains at least one subject data segment. Finally, based on the query word, data query is performed in the subject data segment to be queried to obtain the target query result. By means of offline clustering, combined fragments, and online query, the purpose of reducing network and system load and ensuring retrieval effect is achieved, thereby achieving the technical effect of reducing system resource consumption, reducing query result errors, and improving query speed, thereby solving the technical problems of large resource consumption, large result errors, and slow query speed of the data query method provided by the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0031] Figure 1 It is a schematic diagram of a data query method according to the prior art;

[0032] Figure 2 is a schematic diagram of a data query method according to an embodiment of the present invention;

[0033] Figure 3 is a flow chart of a data query method according to an embodiment of the present invention;

[0034] Figure 4 is a schematic diagram of an optional data storage according to an embodiment of the present invention;

[0035] Figure 5 is a flow chart of an optional data query method according to an embodiment of the present invention;

[0036] Figure 6is a flow chart of an optional offline process of a data query method according to an embodiment of the present invention;

[0037] Figure 7 is a flow chart of an optional online process of a data query method according to an embodiment of the present invention;

[0038] Figure 8 is a schematic diagram of a system architecture of an optional data query method according to an embodiment of the present invention;

[0039] Fig. 9 is a schematic diagram of the system level of an optional data query method according to an embodiment of the present invention;

[0040] Fig.10 It is a structural block diagram of a data query device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0042] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0043] Figure 1 is a schematic diagram of a data query method according to the prior art, such as Figure 1As shown, in the prior art, ES nodes are divided into three categories: data nodes, index nodes, and query nodes. Data nodes are mainly responsible for the storage, inverse query, intersection, union, and sorting of sharded data. The entire data set is divided into shards in the form of indexes and distributed to each data node; index nodes receive data updates, perform preprocessing, and send them to the specified shards for writing according to the routing algorithm; query nodes send query statements to all shards of the index and summarize and reorder the returned results of each shard.

[0044] Still Figure 1 As shown, in the prior art, when querying, the query request is sent to all nodes for retrieval, which results in a large amount of network bandwidth consumption and system load overhead; when updating data, since Lucene maintains a data buffer internally, ES performs a refresh operation at regular intervals to write the buffer data to a new segment and clear the buffer, and each index and shard requires some memory and CPU resources, and each shard runs the search on a single CPU (Central Processing Unit) thread. Since there is a search across a large number of shards when updating data, the search thread pool of the node may be exhausted, resulting in low throughput, slow search speed and cluster instability.

[0045] In order to solve the above problems, an embodiment of the present invention provides a data query method. Figure 2 is a schematic diagram of a data query method according to an embodiment of the present invention. Figure 2 As shown, the embodiment of the present invention trains and maintains a global semantic model, clusters the data set based on the semantic model, and uses the hierarchical mechanism of the ES index to divide the clustered topics into separate shards. When querying, you can select several most relevant shards to query to reduce the number of shards that need to participate in the query, thereby reducing the network and system load. In addition, when the data is updated, the index node pre-processes the incremental data and routes the data to the corresponding shard based on the semantic model, without the need to query all shards, thereby reducing the response time and load of the search process.

[0046] An embodiment of the present invention provides an embodiment of a data query method. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0047] Figure 3 is a flow chart of a data query method according to an embodiment of the present invention. Figure 3 As shown, the method comprises the following steps:

[0048] Step S31, obtaining a query word, wherein the query word is used to determine the data to be queried in the current data set;

[0049] In an optional solution provided in the above step S31, the above query term may include one or more keywords, which may be but are not limited to: data, popular words, expressions, and the above query term may be used to characterize the user's query needs. It is understandable that the above current data set refers to the data set currently contained in the search engine, and the data set may be continuously updated to improve the relevance of the search results. It should also be noted here that the above data to be queried may be the data in the above current data set corresponding to the above query term.

[0050] Specifically, for example, a user enters the query term "Peking University ranking" in the search engine and clicks on the query. The search engine then queries and matches the ranking data corresponding to the keyword "Peking University" in the system's current data set, and then outputs one or more pieces of data to be queried that contain the query term. The user can obtain the required information from the data to be queried.

[0051] Step S32, based on the query word, perform a data query in the sampling pool corresponding to the current data set to obtain a sampling query result, wherein the sampling query result includes at least one target sampling document in the sampling pool that matches the query word;

[0052] In an optional solution provided by the above step S32, the above sampling pool may be a data set related to the above query term in the above current data set. It should also be noted here that the sampling pool may be a sampling pool generated offline in advance. Specifically, in an offline state, based on the historical query results of the above query term or related query terms, a data set corresponding to the above query term may be formed, thereby solving the technical problems of large resource consumption and slow query speed caused by online generation of the sampling pool, thereby improving the user experience.

[0053] In an optional solution provided by the above step S32, it is also necessary to explain that the above at least one target sampled document may be a sampled document containing some or all of the keywords in the above query term, or may be one or more sampled documents that do not contain the above query term but are related to the above query term. It is easy to understand that based on the relevance between the above at least one target sampled document and the above query term, the above sampled query result may be a relevance ranking for the above at least one target sampled document, thereby facilitating the user to obtain the query information most relevant to the query term.

[0054] Specifically, for example, a user enters the query term "Peking University ranking" in the search engine, and there is a sampling pool related to the query term in the data set of the search engine, which may include but is not limited to: sampling documents containing only Peking University's ranking results, sampling documents containing all university ranking data, and sampling documents containing rankings of different majors of Peking University. After clicking the query, the search engine can rank the corresponding query results from high to low according to the relevance and present them to the user. For example, the sampling document containing only Peking University's ranking results with a higher relevance is placed at the top of the query results, and the sampling document containing all university ranking data with a lower relevance is placed at the back of the query results, so as to facilitate users to quickly find the desired query results, thereby improving the user's experience.

[0055] Step S33, using the sample query result, determining the subject data segment to be queried from the clustering result corresponding to the current data set, wherein the clustering result includes multiple data fragments, and each data fragment of the multiple data fragments includes at least one subject data segment;

[0056] In an optional solution provided in the above step S33, the above clustering result can be the result obtained by performing offline clustering on the above current data set based on the semantic model. It should also be noted here that in an offline state, the above current data set is preprocessed based on the semantic model, specifically, the document vector of each document is extracted, and the document vector is divided into multiple topics based on a clustering algorithm (such as the kmeans algorithm), and the topic is used as a single or multiple topic data segments, and then the topic data segments are combined into data slices, written into the index, so as to obtain the clustering result.

[0057] In an optional solution provided by the above step S33, it should also be noted that each of the above multiple data slices can be composed of one or more subject data segments, and the multiple data slices are relatively independent due to the different clustering topics. When querying, the corresponding data slice can be queried based on the topic closest to the above query term. Therefore, the technical problem of high scheduling overhead caused by querying too many data slices can be avoided, thereby reducing the error of the query result, reducing resource consumption and improving the query speed.

[0058] Step S34, based on the query word, perform data query in the subject data segment to be queried to obtain the target query result.

[0059] In an optional solution provided in the above step S34, the above data segment to be queried may be one or more subject data segments related to the above query term. It is understandable that, when querying, the subject data segments corresponding to the query term may be voted and ranked, one or more optimal subject data segments may be selected, and based on the optimal subject data segments, the corresponding data slices may be queried to obtain the above target query result, which may include one or more data corresponding to the above query term.

[0060] The following combination Figure 4 , Figure 5 The above data query method is further explained.

[0061] Figure 4 It is a schematic diagram of an optional data storage according to an embodiment of the present invention. In the ES data storage, the data set corresponding to the index file is divided into multiple shards (i.e., the above-mentioned data shards) in the form of indexes. Each index corresponds to one or more segments (i.e., the above-mentioned subject data segments). Therefore, when querying, one or more segments can be determined based on the index, and then the corresponding shards can be selected in the system without querying all the shards in the index file, thereby reducing the error of the query result, reducing the consumption of system resources, and improving the query speed, thereby improving the user experience.

[0062] Figure 5 is a flow chart of an optional data query method according to an embodiment of the present invention. Figure 5 As shown, in order to reduce the resource consumption of the system, the system can implement semantic clustering offline. Specifically, in the offline state, the system can extract semantic features in the data set, and then perform topic clustering on the data set according to the clustering strategy, and divide the data set into multiple shards to match query results from them when querying.

[0063] Still Figure 5 As shown, when a user makes an online query, he inputs a query word, and the system performs semantic extraction on the input query word and sorts the query results of the query word in the sample by shards. Then, according to the shard selection strategy, the optimal topic data segment corresponding to the query word is determined. Furthermore, based on the shard results generated in the offline state, the data node queries the shard corresponding to the optimal topic data segment and summarizes the query results of the optimal topic data segment.

[0064] In an embodiment of the present invention, a query word is first obtained, wherein the query word is used to determine the data to be queried in the current data set. Then, based on the query word, data query is performed in the sampling pool corresponding to the current data set to obtain a sampling query result, wherein the sampling query result includes at least one target sampling document in the sampling pool that matches the query word. Then, using the sampling query result, the subject data segment to be queried is determined from the clustering result corresponding to the current data set, wherein the clustering result includes multiple data fragments, and each data fragment of the multiple data fragments contains at least one subject data segment. Finally, based on the query word, data query is performed in the subject data segment to be queried to obtain the target query result. By means of offline clustering, combined fragments, and online query, the purpose of reducing network and system load and ensuring retrieval effect is achieved, thereby achieving the technical effect of reducing system resource consumption, reducing query result errors, and improving query speed, thereby solving the technical problems of large resource consumption, large result errors, and slow query speed of the data query method provided by the related technology.

[0065] The above method of the above embodiment of the present invention is further introduced below.

[0066] In an optional embodiment, the data query method further includes:

[0067] Step S35, optionally, randomly sampling the current data set to obtain a sampling pool, wherein the sampling pool includes a plurality of candidate sampling documents;

[0068] Step S36, semantic clustering is performed on multiple documents in the current data set to obtain a clustering result.

[0069] In the optional solution provided by the present invention, a pre-trained model is maintained in an offline state, and the model can be but is not limited to: a BERT model. The model is fine-tuned using the current data set in the system, and then the data set is randomly sampled, and multiple candidate sampling documents are combined into a sampling pool, wherein the multiple candidate sampling documents can correspond to multiple topics in the current data set, and based on the multiple topics, the multiple candidate sampling documents can be semantically clustered to obtain clustering results of multiple topics.

[0070] In an optional embodiment, in step S32, based on the query word, data query is performed in the sampling pool corresponding to the current data set, and the sample query result obtained includes:

[0071] Step S321, calculating the matching degree between the query word and multiple candidate sampling documents in the sampling pool to obtain a matching order;

[0072] Step S322: Select at least one target sampling document in the sampling query result from the candidate sampling documents based on the matching order.

[0073] In the optional solution provided by the present invention, during online query, based on the input query word, the matching degree between the query word and multiple candidate sampled documents in the sampling pool is calculated, and then based on the matching degree, the sampled documents are ranked to obtain a matching order, wherein the matching order can be sorted from high to low according to the matching degree. Further, based on the matching order of the sampled documents, the sampled document with the highest matching degree can be selected as the target sampled document.

[0074] In the above optional implementation, the technical effect that can be achieved is: based on the query terms, matching sample documents are determined from the system, and the degree of matching between the two is determined, so that users can select the desired query results from them, thereby improving the relevance of the query results to the query terms and enhancing the user experience.

[0075] In an optional implementation, in step S36, semantic clustering is performed on multiple documents in the current data set, and the clustering results obtained include:

[0076] Step S361, using a document semantic model, extracting features from each of the multiple documents in the current data set to obtain a feature vector for each of the multiple documents, wherein the document semantic model is a neural network model obtained by offline training based on the current data set using an attention mechanism;

[0077] Step S362: semantically cluster the current data set based on the feature vector to obtain a clustering result.

[0078] In the optional solution provided in the above step S361, the above document semantic model can be used to pre-train the data set in the search engine, and the document semantic model can include but is not limited to: BERT model, TFIDF model, RNN model, LSTM model.

[0079] The following combination Figure 6 The above data query method is further explained.

[0080] Figure 6 is a flowchart of an optional offline process of a data query method according to an embodiment of the present invention. Figure 6 As shown, based on the existing documents in the search engine, a semantic model can be established, and the semantic model can be a neural network model based on the attention mechanism shown in the following formula (1):

[0081]

[0082] It should also be noted here that the neural network model can be composed of 12 layers of Transformers encoders, each with a hidden size of 768. For a document containing n feature tags, after feature extraction of the document, n feature vectors with a dimension of 768 can be obtained, and the output vectors of all words in the document can be arranged into a matrix of size n×768. Based on the feature vector, semantic clustering of existing documents can be performed.

[0083] In an optional implementation, in step S362, semantic clustering is performed on the current data set based on the feature vector, and the clustering results obtained include:

[0084] Step S3621, based on the feature vector of each document in the multiple documents, a preset clustering algorithm is used to semantically cluster the multiple documents in the current data set to obtain multiple topic data segments, wherein the topic centroid of each topic data segment in the multiple topic data segments corresponds to a semantic clustering center;

[0085] Step S3622, shard and combine multiple subject data segments to obtain multiple data shards in the clustering results.

[0086] In the optional solution provided in the above step S3621, the above preset clustering algorithm may include but is not limited to: kmeans algorithm, dbscan algorithm. The above topic centroid may correspond to a semantic clustering center of the above multiple topic data segments, which may be used to divide data fragments.

[0087] Still Figure 6 As shown in the figure, assume that the feature vector of each document is semantically clustered according to the kmeans algorithm. First, the document is divided into k topics, and the topic centroids are recorded and stored. Specifically, the data set x = {x1, x2, ..., x N} is divided into k categories:

[0088]

[0089] x={x1,x2,…,x n} Formula (3)

[0090] In the above formulas (2) and (3), the objective function J is the data point x n and the center of mass μ k The sum of squares is used to minimize the objective function by the calculation method shown in the following formulas (4) and (5).

[0091]

[0092] Still Figure 6As shown, after the above semantic clustering is performed on the data set in the search engine, the topic is taken as a single or multiple topic data segments, and the single or multiple topic data segments are combined into l data fragments (l≤k) and written into the ES index.

[0093] In the above optional implementation, the technical effect that can be achieved is: in an offline state, semantic models and clustering algorithms are used to perform semantic clustering on existing data sets in the search engine, so as to facilitate online queries based on query terms and quickly and accurately match corresponding data fragments, thereby improving the relevance of query results to query terms and enhancing the user experience.

[0094] In an optional implementation, the data query method provided in step S362 above further includes:

[0095] Step S3623, using the document semantic model to perform feature extraction on the newly added document in the current data set, and extracting the feature vector of the newly added document;

[0096] Step S3624, determining a topic data segment to be added from the multiple topic data segments based on the feature distance between the feature vector of the newly added document and the topic centroid of each topic data segment in the multiple topic data segments;

[0097] Step S3625, adding the newly added document to the subject data segment to be added.

[0098] In the optional solution provided in the above step S3623, the above newly added document may be an updated document corresponding to the above query term, and the updated document may include incremental data corresponding to the query term.

[0099] Still Figure 6 As shown, when the data set in the search engine is updated, the feature vector of the newly added document is extracted based on the above document semantic model, and the feature distance between the feature vector of the newly added document and the topic centroid of each topic data segment is determined based on the above formula (2), and the topic data segment with the smallest feature distance is determined as the topic data segment to be added. Furthermore, the newly added document is added to the topic data segment to be added to obtain an updated data set.

[0100] In an optional implementation, in step S3625, adding the newly added document to the added topic data segment includes:

[0101] Step S36251, writing the newly added document and the corresponding subordinate information into the buffer of the current data set, wherein the subordinate information is used to describe the subject data segment to be added corresponding to the newly added document and the data shard to be added to which the subject data segment to be added belongs;

[0102] Step S36252: Based on the subordinate information, merge the newly added document into the subject data segment to be added in the data slice to be added.

[0103] Still Figure 6 As shown, after determining the optimal topic data segment of the newly added document based on the feature vector of the newly added document, the newly added document and the corresponding subordinate information are recorded and stored, and based on the routing information contained in the subordinate information, the to-be-added topic data segment corresponding to the newly added document is added to the target data shard, and the information of the newly added document is written into the shard buffer. The system periodically refreshes the shard buffer to create a new topic data segment. At the same time, the system periodically performs topic data segment merging to merge the new topic data segment into the existing topic data segment. Based on the updated topic data segment, the system can periodically rebuild the index. In addition, based on the updated data set, the system can re-perform semantic clustering to update the system data.

[0104] In the above optional implementation, the technical effect that can be achieved is: in an offline state, the semantic model and clustering algorithm are used to semantically cluster the existing data sets in the search engine and update them in a timely manner, so that when querying online, the latest query results can be quickly and accurately matched based on the query terms, thereby improving the relevance of the query results and the query terms and enhancing the user experience. At the same time, the subject data segments are combined into large data fragments, which reduces the resources occupied by the data fragments and improves the query speed of the system and the stability of the cluster.

[0105] In an optional implementation, in step S33, using the sample query result, determining the subject data segment to be queried from the clustering result corresponding to the current data set includes:

[0106] Step S331, determining the candidate topic data segment to which each target sampled document in the sampled query result belongs from the clustering result;

[0107] Step S332, based on the sample query results and the matching order, weighted voting is performed on the candidate topic data segments to obtain voting results;

[0108] Step S333, in response to the voting result satisfying the preset condition, determining the subject data segment to be queried according to the voting result.

[0109] As one of the specific implementations of the above embodiment of the present invention, the above embodiment is applied to the news search application scenario in the field of big data technology. Taking this scenario as an example, Figure 7 , Figure 8 and Fig. 9 The data query process shown further illustrates the data query method provided by the above optional implementation mode of the present invention.

[0110] Figure 7is a flowchart of an optional online process of a data query method according to an embodiment of the present invention. Figure 7 As shown, when a user enters a query word when searching online, the system first pre-processes the query word, and the pre-processing process may include but is not limited to: word segmentation and punctuation removal, and then obtains the query results of the query word in the shard sampling. The shard sampling process is used to determine the sample documents corresponding to the query word.

[0111] Specifically, based on the query sampling method, the description information of each subject data segment is obtained, a queries are randomly selected from the query log as the initial query words, and in each round of retrieval, the first b documents of the retrieval results returned by each collection are added to the sample collection, and 1 keyword is randomly selected from the sample collection documents to add to the candidate query words. Sampling is stopped when the number of documents in the sample collection is greater than or equal to 3% of the total number of documents in all collections.

[0112] Still Figure 7 As shown in the figure, based on the query words, the corresponding sampled query results are obtained from the sample documents obtained by shard sampling. Furthermore, based on the shard selection strategy, one or more optimal topic data segments are selected from them. The query node broadcasts the query information to the data node, and then the data node queries the data shards in the system. During the query process, the unselected topic data segments can be skipped based on the query information. Specifically, Figure 4 As shown, when the query shard corresponding to the query term is "sub-shard 1-2", since the query information only contains information related to "sub-shard 1-2", during the query, data shards such as "sub-shard 1-1" can be skipped, thereby reducing the query response time and improving the query speed.

[0113] Still Figure 7 As shown, the data nodes can summarize the query results of the subject data segment, and the query nodes can summarize the query results of the data shards, so as to obtain the query results corresponding to the query terms. Specifically, for the sampling pool, the documents in the query result set are first ranked, and based on the ranking order, the subject data segments to which they belong are voted. The voting score process for each ranking can be shown as the following formula (6):

[0114] S(i)=D - h×V i Formula (6)

[0115] In the above formula (6), V i represents the score of the sample query result, D represents an optional parameter, h represents the ranking of the query result, and the voting weight index of each ranking shows a decreasing trend. When the voting score is less than the preset threshold, the voting stops, and based on the current voting results, the target topic data segment corresponding to the query term can be determined.

[0116] In the above optional implementation, the technical effect that can be achieved is: during online query, candidate subject data segments can be matched based on the semantic clustering results, and the query results can be voted on based on the ranking of the query results. Ultimately, the subject data segment most relevant to the query term can be determined, thereby improving the relevance of the query results to the query terms and improving the query speed.

[0117] Figure 8 is a schematic diagram of a system architecture of an optional data query method according to an embodiment of the present invention, such as Figure 8 As shown in the figure, the system management platform includes monitoring configuration, sharding configuration, and policy configuration. The system management platform can provide scheduling services for the ES cluster. Incremental data corresponds to a series of task queues for executing data update tasks, which can provide preprocessing services for the ES cluster. When a user queries online, he can enter a query term to initiate a query service, and the ES cluster can complete the query service and output the query results. In addition, the system's preprocessing service, scheduling service, and semantic model service can complete semantic clustering offline.

[0118] Fig. 9 is a schematic diagram of a system level of an optional data query method according to an embodiment of the present invention, such as Fig. 9 As shown in the figure, the configuration layer includes three parts: monitoring configuration, sharding configuration, and policy configuration. Among them, the monitoring configuration is responsible for monitoring the traffic load and other information of the current system. It can determine the distribution of shards based on the historical load conditions to make the load distribution more uniform, and dynamically enable or disable the degradation strategy based on the load conditions; the sharding configuration can be used to determine the distribution and sharding of the current subject segment data; the policy configuration can be used to determine the current sampling library selection and algorithm parameter configuration.

[0119] Still Fig. 9 As shown, the service layer includes preprocessing service, query service, semantic model service, and scheduling service, which are used to encapsulate peripheral functional modules in the form of microservices. Among them, the preprocessing service can be used to preprocess documents and query terms, request the semantic model service to obtain document vectors, and fine-tune the model; the query service can be used to create DSL (Domain-Specific Language) and request query nodes to return results; the semantic model service can be used to encapsulate and call deep learning models; the scheduling service can be responsible for interaction with the configuration platform and trigger operations for periodic redistribution;

[0120] Still Fig. 9As shown, the semantic layer includes feature extraction, document clustering, and shard selection, which facilitates the system to understand the document semantics. Feature extraction can be used to extract document features during semantic clustering; document clustering can be used to cluster documents based on clustering algorithms; shard selection can be used to determine the data shards and subject data segments corresponding to the query terms during online queries.

[0121] Still Fig. 9 As shown, the index layer includes three parts: topic segment merging, document redistribution, and shard sampling. Among them, topic segment merging can be used to merge new topic data segments into whole segments when updating data; document redistribution can be used to rebuild the index regularly; shard sampling can be used to determine the sample documents corresponding to the query terms, so as to further determine the query results of the query terms.

[0122] In the above optional implementations, the following technical effects can be achieved: (1) In emergency scenarios, a distributed search system can be used to optimize the load, and offline clustering can be used to make similar data present a spatially local distribution, thereby reducing the error of search results; (2) By utilizing the segmented structure, the subject data segments are combined into large data shards, which reduces the resources occupied by the data shards and ensures the normal scheduling of the data shards and the stability of the cluster; (3) The subject distribution is modeled through offline sampling, and the content and number of the query data shards are dynamically determined, thereby solving the problem of shard selection; (4) Through model fine-tuning and re-clustering, the semantic model is adaptively updated to ensure the real-time nature of the data set.

[0123] In the present embodiment, a data query device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, a "module" is a combination of software and / or hardware that can implement a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0124] Fig.10 is a structural block diagram of a data query device according to an embodiment of the present invention. Fig.10 As shown, the device comprises:

[0125] An acquisition module 1001 is used to acquire a query word, wherein the query word is used to determine the data to be queried in the current data set;

[0126] The first query module 1002 is used to perform data query in the sampling pool corresponding to the current data set based on the query word to obtain a sampling query result, wherein the sampling query result includes at least one target sampling document in the sampling pool that matches the query word;

[0127] A determination module 1003 is used to determine the subject data segment to be queried from the clustering result corresponding to the current data set using the sample query result, wherein the clustering result includes multiple data fragments, and each data fragment of the multiple data fragments includes at least one subject data segment;

[0128] The second query module 1004 is used to perform data query in the subject data segment to be queried based on the query word to obtain the target query result.

[0129] Optionally, the above data query method also includes: a clustering module 1005, which is used to randomly sample the current data set to obtain a sampling pool, wherein the sampling pool includes multiple candidate sampling documents; and semantically cluster multiple documents in the current data set to obtain a clustering result.

[0130] Optionally, the first query module 1002 is also used to: based on the query term, perform data query in the sampling pool corresponding to the current data set, and obtain the sampling query result including: calculating the matching degree between the query term and multiple candidate sampling documents in the sampling pool to obtain a matching order; based on the matching order, select at least one target sampling document in the sampling query result from the candidate sampling documents.

[0131] Optionally, the clustering module 1005 is also used to: perform semantic clustering on multiple documents in the current data set, and obtain clustering results including: using a document semantic model to extract features from each of the multiple documents in the current data set to obtain a feature vector for each of the multiple documents, wherein the document semantic model is a neural network model obtained by offline training based on the current data set using an attention mechanism; based on the feature vector, perform semantic clustering on the current data set to obtain a clustering result.

[0132] Optionally, the clustering module 1005 is also used to: perform semantic clustering on the current data set based on the feature vector, and obtain the clustering results including: based on the feature vector of each document in the multiple documents, perform semantic clustering on the multiple documents in the current data set using a preset clustering algorithm to obtain multiple topic data segments, wherein the topic centroid of each topic data segment in the multiple topic data segments corresponds to a semantic clustering center; perform fragmentation and combination on the multiple topic data segments to obtain multiple data fragments in the clustering results.

[0133] Optionally, the clustering module 1005 is also used to: utilize the document semantic model to perform feature extraction on the newly added documents in the current data set, and extract the feature vector of the newly added documents; determine the topic data segment to be added from multiple topic data segments based on the feature distance between the feature vector of the newly added documents and the topic centroid of each topic data segment in multiple topic data segments; and add the newly added documents to the topic data segment to be added.

[0134] Optionally, the clustering module 1005 is also used to: add the newly added document to the added subject data segment, including: writing the newly added document and the corresponding subordinate information into the buffer of the current data set, wherein the subordinate information is used to describe the subject data segment to be added corresponding to the newly added document and the data shard to be added to which the subject data segment to be added belongs; according to the subordinate information, merge the newly added document into the subject data segment to be added in the data shard to be added.

[0135] Optionally, the above-mentioned determination module 1003 is also used to: use the sampling query results to determine the subject data segment to be queried from the clustering results corresponding to the current data set, including: determining the candidate subject data segment to which each target sampling document in the sampling query results belongs from the clustering results; based on the sampling query results and the matching order, performing weighted voting on the candidate subject data segments to obtain the voting results; in response to the voting results satisfying preset conditions, determining the subject data segment to be queried according to the voting results.

[0136] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0137] According to another aspect of an embodiment of the present invention, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the aforementioned data query methods.

[0138] Optionally, in this embodiment, the memory may be configured as a program for executing the following steps:

[0139] Step S1, obtaining a query word, wherein the query word is used to determine the data to be queried in the current data set;

[0140] Step S2, based on the query word, perform data query in the sampling pool corresponding to the current data set to obtain a sampling query result, wherein the sampling query result includes at least one target sampling document in the sampling pool that matches the query word;

[0141] Step S3, using the sample query result, determining the subject data segment to be queried from the clustering result corresponding to the current data set, wherein the clustering result includes multiple data fragments, and each data fragment of the multiple data fragments includes at least one subject data segment;

[0142] Step S4, based on the query word, perform data query in the subject data segment to be queried to obtain the target query result.

[0143] Optionally, in this embodiment, the above-mentioned memory may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0144] Optionally, in this embodiment, the processor may be configured as a program for executing the following steps:

[0145] Step S1, obtaining a query word, wherein the query word is used to determine the data to be queried in the current data set;

[0146] Step S2, based on the query word, perform data query in the sampling pool corresponding to the current data set to obtain a sampling query result, wherein the sampling query result includes at least one target sampling document in the sampling pool that matches the query word;

[0147] Step S3, using the sample query result, determining the subject data segment to be queried from the clustering result corresponding to the current data set, wherein the clustering result includes multiple data fragments, and each data fragment of the multiple data fragments includes at least one subject data segment;

[0148] Step S4, based on the query word, perform data query in the subject data segment to be queried to obtain the target query result.

[0149] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0150] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0151] In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0152] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0153] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0154] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0155] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data query method, characterized in that: include: Obtaining a query word, wherein the query word is used to determine the data to be queried in the current data set; Randomly sampling the current data set to obtain a sampling pool, wherein the sampling pool includes a plurality of candidate sampling documents; Performing offline semantic clustering on multiple documents in the current data set to obtain a clustering result; Calculating the matching degree between the query word and the plurality of candidate sampling documents in the sampling pool to obtain a matching order; Based on the matching order, selecting a sampling query result from the candidate sampling documents, wherein the sampling query result includes at least one target sampling document matching the query term; Determining, from the clustering result, a candidate topic data segment to which each target sampled document in the sampled query result belongs, wherein the clustering result includes a plurality of data fragments, and each of the plurality of data fragments includes at least one topic data segment; Based on the sample query results and the matching order, weighted voting is performed on the candidate subject data segments to obtain voting results; In response to the voting result satisfying a preset condition, determining the subject data segment to be queried according to the voting result; Based on the query words, a data query is performed in the subject data segment to be queried to obtain a target query result.

2. The data query method according to claim 1, characterized in that: Performing semantic clustering on the multiple documents in the current data set to obtain the clustering result includes: Using a document semantic model, extracting features from each of the multiple documents in the current data set to obtain a feature vector for each of the multiple documents, wherein the document semantic model is a neural network model obtained by offline training based on the current data set using an attention mechanism; Based on the feature vector, semantic clustering is performed on the current data set to obtain the clustering result.

3. The data query method according to claim 2, characterized in that: Based on the feature vector, semantic clustering is performed on the current data set to obtain the clustering result including: Based on the feature vector of each document in the multiple documents, a preset clustering algorithm is used to semantically cluster the multiple documents in the current data set to obtain multiple topic data segments, wherein the topic centroid of each topic data segment in the multiple topic data segments corresponds to a semantic cluster center; The multiple subject data segments are fragmented and combined to obtain the multiple data fragments in the clustering result.

4. The data query method according to claim 3, characterized in that: The method further comprises: Using the document semantic model to perform feature extraction on the newly added document of the current data set, and extracting a feature vector of the newly added document; Determining a to-be-added topic data segment from the multiple topic data segments based on a feature distance between a feature vector of the newly added document and a topic centroid of each topic data segment in the multiple topic data segments; Add the newly added document to the subject data segment to be added.

5. The data query method according to claim 4, characterized in that: Adding the newly added document to the added subject data segment includes: Writing the newly added document and the corresponding subordinate information into the buffer of the current data set, wherein the subordinate information is used to describe the subject data segment to be added corresponding to the newly added document and the data shard to be added to which the subject data segment to be added belongs; According to the subordinate information, the newly added document is merged into the to-be-added subject data segment in the to-be-added data slice.

6. A data query device, characterized in that: include: An acquisition module, used to acquire a query word, wherein the query word is used to determine the data to be queried in the current data set; A clustering module, configured to randomly sample the current data set to obtain a sampling pool, and to perform offline semantic clustering on a plurality of documents in the current data set to obtain a clustering result, wherein the sampling pool includes a plurality of candidate sampling documents; A first query module, configured to calculate a matching degree between the query word and the plurality of candidate sampling documents in the sampling pool to obtain a matching order, and select a sampling query result from the candidate sampling documents based on the matching order, wherein the sampling query result includes at least one target sampling document matching the query word; a determination module, configured to determine, from the clustering result, a candidate topic data segment to which each target sampled document in the sampled query result belongs, and to perform weighted voting on the candidate topic data segments based on the sampled query result and the matching order to obtain a voting result, and in response to the voting result satisfying a preset condition, determine a topic data segment to be queried according to the voting result, wherein the clustering result includes a plurality of data fragments, and each of the plurality of data fragments includes at least one topic data segment; The second query module is used to perform data query in the subject data segment to be queried based on the query word to obtain a target query result.

7. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data query method according to any one of claims 1 to 5.

Images