Index construction method, field search method, device, equipment and medium

Abstract

The present application is applicable to the field of data processing technology, and in particular, to an index construction method, a field search method, a device, and a medium. The method performs field extraction on any data column in a public data set according to the field extraction method corresponding to the data type of the data column, obtains the field features and feature vectors of the data column, constructs an inverted index based on the field features and storage information, and constructs a vector index based on the feature vector and storage information. When querying, a first data set is determined based on the data to be queried and the inverted index, a second data set is determined based on the vector to be queried and the vector index, a candidate data set is formed based on the first data set and the second data set, and all data in the candidate data set is sorted based on the data to be queried, and the sorted data is obtained as the query result. A more refined index structure is constructed in the public data set, which improves the comprehensiveness and accuracy of the search results.

Description

Technical Field

[0001] The present application is applicable to the field of data processing technology, and in particular relates to an index construction method, a field search method, an apparatus, a device and a medium. Background Art

[0002] In the scenario of government data opening, in order to give full play to the value of open data, the open platform provides users with search functions for massive data. At present, traditional data search technology mainly searches and matches in open data through shallow information such as dataset name, file name and dataset type. Although this method can achieve rapid data positioning to a certain extent, its search results are often limited to data information directly related to search keywords such as dataset name, and cannot provide more refined search, making the search results relatively single and lacking in depth and breadth. In addition, government open data often involves multiple departments and business fields, and there are often correlations between these open data. Traditional search technology often cannot obtain other data information indirectly related to the query, making it impossible for users to obtain comprehensive and coherent data information, which further limits the value of open data. Therefore, how to provide users with refined search and improve the accuracy of data search has become an urgent problem to be solved. Summary of the invention

[0003] In view of this, the embodiments of the present application provide an index construction method, a field search method, an apparatus, a device and a medium to solve the problem of how to provide users with refined searches and improve the accuracy of data searches.

[0004] In a first aspect, an embodiment of the present application provides an index construction method, the index construction method comprising:

[0005] Acquire a public data set, and for any data column in the public data set, perform field extraction on the data column according to a field extraction method corresponding to the data type of the data column, obtain field features of the data column, and determine a feature vector corresponding to each field feature;

[0006] The storage information of each data column in the public data set is obtained, an inverted index is constructed according to the field characteristics and storage information of each data column, and a vector index is constructed according to the feature vector and storage information of each data column.

[0007] In a second aspect, an embodiment of the present application provides a field search method. After an inverted index and a vector index are obtained by using the index construction method described in the first aspect, the field search method includes:

[0008] Acquire the data to be queried, and determine the vector to be queried of the data to be queried;

[0009] Determine a first data set according to the data to be queried and the inverted index, and determine a second data set according to the vector to be queried and the vector index;

[0010] forming a candidate data set according to the first data set and the second data set;

[0011] According to the data to be queried, all the data in the candidate data set are sorted, and the sorted data obtained is the query result.

[0012] In a third aspect, an embodiment of the present application provides an index construction device, the index construction device comprising:

[0013] A feature extraction module is used to obtain a public data set, and for any data column in the public data set, perform field extraction on the data column according to the field extraction method corresponding to the data type of the data column, obtain the field features of the data column, and determine the feature vector corresponding to each field feature;

[0014] The index building module is used to obtain the storage information of each data column in the public data set, build an inverted index according to the field characteristics and storage information of each data column, and build a vector index according to the feature vector and storage information of each data column.

[0015] In a fourth aspect, an embodiment of the present application provides a field search device. After obtaining an inverted index and a vector index by adopting the index construction method described in the first aspect, the field search device includes:

[0016] A query acquisition module, used to acquire the data to be queried and determine the vector to be queried of the data to be queried;

[0017] A first determination module, configured to determine a first data set according to the data to be queried and the inverted index, and to determine a second data set according to the vector to be queried and the vector index;

[0018] a second determining module, configured to form a candidate data set according to the first data set and the second data set;

[0019] The result acquisition module is used to sort all the data in the candidate data set according to the data to be queried, and obtain the sorted data as the query result.

[0020] In a fifth aspect, an embodiment of the present application provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the index building method as described in the first aspect or the field search method as described in the second aspect is implemented.

[0021] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the index construction method as described in the first aspect, or the field search method as described in the second aspect is implemented.

[0022] Compared with the prior art, the embodiments of the present application have the following beneficial effects: when constructing an index, the present application uses a field extraction method corresponding to the data type of the data column for any data column in the public data set, extracts the field of the data column, obtains the field features and feature vectors of the data column, constructs an inverted index based on the field features and storage information of each data column, and constructs a vector index based on the feature vector and storage information of each data column. When querying, the first data set is determined based on the query data and the inverted index, the second data set is determined based on the query vector and the vector index, a candidate data set is formed based on the first data set and the second data set, and all the data in the candidate data set is sorted based on the data to be queried, and the sorted data is obtained as the query result. Among them, according to the field features, feature vectors and storage information of the data column in the public data set, the inverted index and vector index are respectively constructed, and a more refined index structure in the public data set is constructed, so that when querying, not only can the information matching the query data be quickly searched according to the constructed inverted index and vector index, but also a more refined search can be performed within the public data set, thereby improving the comprehensiveness and accuracy of the search results. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1 This is a schematic diagram of an application environment of an index building method and a field search method provided in Example 1 of the present application;

[0025] Figure 2 This is a flowchart of an index construction method provided in Example 2 of the present application;

[0026] Figure 3 It is a flowchart of an index construction method provided in Example 3 of the present application;

[0027] Figure 4 It is a flowchart of an index construction method provided in Example 4 of the present application;

[0028] Figure 5This is a flowchart of a field search method provided in Embodiment 5 of the present application;

[0029] Figure 6 This is a flowchart of a field search method provided in Example 6 of the present application;

[0030] Figure 7 This is a flowchart of a field search method provided in Embodiment 7 of the present application;

[0031] Figure 8 It is a structural schematic diagram of an index building device provided in Embodiment 8 of the present application;

[0032] Fig. 9 It is a structural diagram of a field search device provided in Embodiment 9 of the present application;

[0033] Fig.10 It is a structural diagram of a computer device provided in Example 10 of the present application. DETAILED DESCRIPTION

[0034] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0035] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0036] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0037] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0038] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0039] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0040] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0041] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0042] It should be understood that the size of the serial numbers of the steps in the following embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0043] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0044] The index construction method and field search method provided in the first embodiment of the present application can be applied to Figure 1In the application environment, the server communicates with the client, the server provides index building and field search services, and the client triggers index building and field search tasks to the server. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, personal digital assistants (PDAs), and other devices. The computer device corresponding to the server can be implemented with an independent server or a server cluster consisting of multiple servers.

[0045] See also Figure 2 , is a flow chart of an index construction method provided in Example 2 of the present application, and the index construction method is applied to Figure 1 The server in the example connects to the client to obtain the public data set sent by the client. Figure 2 As shown, the index building method may include the following steps:

[0046] Step S201, obtain a public data set, perform field extraction on any data column in the public data set according to the field extraction method corresponding to the data type of the data column, obtain field features of the data column, and determine a feature vector corresponding to each field feature.

[0047] In an embodiment of the present application, a data set may refer to a collection of text data, which can be used to store and organize text-type data, wherein the data in the data set may be stored in a table or file, etc. A public data set may refer to a data set that can be shared and used. For a public data set whose data storage form is a table, each column in the storage table represents a different feature or variable, which is used to describe information about specific aspects of each sample in the data set. Each column in the storage table is a data column. For example, in a public data set whose data storage form is a table, a column in the data set may be a data column representing age features, or a data column representing gender features, etc. A field extraction method may refer to a technique for extracting important features from data in a data column. A field feature may refer to data representing important features of a data column. A feature vector may refer to a vector representation of a field feature.

[0048] In view of the different features or variables represented by each data column in the public data set, the data columns can be divided into enumeration type data columns, numerical type data columns, text type data columns, etc. according to the data type of the data column. For example, an enumeration type data column may refer to a data column representing data with values ​​of a fixed set of values ​​such as gender features, grade features, or status features; a numerical type data column may refer to a data column representing age features, score features, number features, etc., which are composed of numerical values; a text type data column may refer to a data column representing name features, address features, description features, etc., which are composed of text.

[0049] Specifically, in the process of determining the field characteristics of an enumeration type data column, the number of occurrences of each enumeration value in the enumeration type data column can be counted. If the number of occurrences of the enumeration value is greater than a predefined threshold, the enumeration value is determined to be the field characteristic of the enumeration type data column; in the process of determining the field characteristics of a numerical type data column, the minimum value calculation function, the maximum value calculation function and the quantile calculation function can be used to calculate the minimum value, maximum value, 5% quantile, 15% quantile, 50% quantile and 95% quantile of the data column as the field characteristics of the numerical type data column, for example, in the Python programming language, the minimum value of the data column can be calculated by the np.min() function, and the maximum value of the data column can be calculated by the np.max() function. The np.percentile() function calculates the quantile; in the process of determining the text type data column, for meaningful text data composed of string data such as numbers, letters or Chinese characters, the key fields of the text data can be extracted through entity recognition technology and relationship extraction technology, and fine-grained type recognition can be performed to obtain the field characteristics of the text type data column. For meaningless text data composed of string data such as numbers, letters or Chinese characters, the text data can be divided into subsequences, and the word frequency-inverse document frequency value of each subsequence in the data column is calculated, which is used as the importance score of each subsequence in the data column, and the subsequences with importance scores greater than a pre-defined threshold are extracted as the field characteristics of the text type data column.

[0050] In the process of determining the feature vectors corresponding to the field features of the enumeration type data column, the numerical type data column and the text type data column, the field features extracted from the data columns of each data type can be converted into a high-dimensional vector representation through a model trained in advance for vector conversion, that is, the feature vector corresponding to each field feature can be obtained.

[0051] Step S202, obtaining storage information of each data column in the public data set, constructing an inverted index according to the field characteristics and storage information of each data column, and constructing a vector index according to the feature vector and storage information of each data column.

[0052] In an embodiment of the present application, the storage information may refer to the storage location of a data column in a public data set. For example, the storage information of the data column may be stored in a public data set numbered 2, or further, the storage information of the data column may be stored in a public data set numbered 2, and its corresponding data column numbered 3. The inverted index may refer to an index structure for storing a mapping relationship between field features and storage information of the field features in a public data set. It may include key values ​​and arrays corresponding to the key values. The key values ​​may be used to store inverted index items such as words or phrases. The arrays corresponding to the key values ​​may be used to store storage information corresponding to the words or phrases in the inverted index items. The vector index may refer to an index structure established based on a feature vector. For example, the vector index may be a Faiss index.

[0053] Specifically, in the process of building an inverted index, first, the storage information of each data column in the public data set is obtained, and the storage information is the storage information of the corresponding field features in the data column, and the dictionary and inverted list in the inverted index structure are respectively built according to the index building tool, and then, the field features of each data column are segmented to obtain each word or phrase corresponding to the field feature after the segmentation processing, and finally, each word or phrase after the segmentation processing is written as a key value into the blank key value corresponding to the dictionary, and a unique identifier is assigned to each word or phrase in the dictionary, and the storage information corresponding to each word or phrase in the dictionary is respectively written into the array of the key value corresponding to the inverted list. In the process of building a vector index, the feature vector and the storage information can be written into a predetermined vector index structure to obtain a vector index.

[0054] The embodiment of the present application obtains a data column for any data type in a public data set, uses a field extraction method corresponding to the data type, extracts fields from the data column, obtains field features and feature vectors of the data column, constructs an inverted index based on the field features and storage information of each data column, and constructs a vector index based on the feature vector and storage information of each data column. A more refined index structure within the public data set is constructed, so that when querying, not only can information matching the query data be quickly searched based on the constructed inverted index and vector index, but a more refined search can also be performed within the public data set, thereby improving the comprehensiveness and accuracy of the search results.

[0055] See also Figure 3 , is a flow chart of an index construction method provided in Example 3 of the present application. Figure 3 As shown, after obtaining the field features and feature vectors of the data column in the above step S201, the index construction method may further include the following steps:

[0056] Step S301 , calculating the similarity of the feature vectors of any two data columns to obtain a similarity calculation result. If the similarity calculation result is greater than a similarity threshold, the two data columns are determined to be a suspected matching pair.

[0057] In an embodiment of the present application, similarity may refer to the distance between two feature vectors in the Euclidean space, the similarity calculation result may refer to the distance between the feature vectors of two data columns in the Euclidean space, the similarity threshold may refer to a predefined similarity value, and a suspected matching pair may refer to two potentially matching data columns whose similarity calculation results are greater than the similarity threshold.

[0058] Specifically, the Cosine similarity calculation formula can be used to calculate the Cosine similarity between the feature vectors of any two data columns in the public data set to obtain a similarity calculation result, and the similarity calculation result of any two data columns can be compared with the similarity threshold. If the similarity calculation result is greater than the similarity threshold, the two data columns are determined to be suspected matching pairs, and the suspected matching pairs in the public data set are recalled through the similarity link method.

[0059] Step S302 , extracting a preset number of first data from the first data columns in the suspected matching pairs, matching all the first data with the second data columns in the suspected matching pairs, and obtaining a matching result.

[0060] In an embodiment of the present application, the first data column may refer to any data column in a suspected matching pair, the second data column may refer to another data column in the suspected matching pair except the first data column, the first data may refer to the data in the first data column, the preset number may refer to a preset number of extractions of the first data, and the matching result may refer to the matching result between the first data and the second data column in the suspected matching pair, which may include both matching and non-matching results.

[0061] Specifically, first, a preset number of first data can be extracted from the first data column in the suspected matching pair through a bilateral sampling method, and then the first data can be matched with the data in the second data column through predefined operators, such as an equal sign, a comparison symbol, and an inclusion relationship. If the first data matches any data in the second data column, the matching result of the first data and the second data column is determined to be a match.

[0062] Step S303, counting the proportion of the first data in the preset number of matched results, if the proportion exceeds a proportion threshold, it is determined that the suspected match pair has passed the verification, and the field features of the first data column and the second data column are associated with each other.

[0063] Step S304, after the first data column and the second data column are written into the inverted index, the association information is written into the array of the first data column and the second data column in the inverted index.

[0064] In an embodiment of the present application, the proportion threshold may refer to a pre-set proportion value, and the association information may refer to information that identifies the association between the field characteristics of the first data column and the second data column. For example, the association information may refer to an identifier that matches the field characteristics of the first data column and the second data column, a link or address associated between the first data column and the second data column, etc.

[0065] Specifically, first, according to the matching results of each first data and second data column calculated in step S302, the number of first data with matching results can be determined, and the proportion of the number of first data with matching results in the preset number can be calculated. Then, if the proportion exceeds the proportion threshold, it is determined that the suspected matching pair verification has passed, the field characteristics of the first data column and the second data column are used to form association information, and the first data column and the second data column are determined to be a data column pair. Finally, after the field characteristics and storage information of the first data column and the second data column are written into the inverted index in step S201, the association information is written into the first data column and the second data column in the array corresponding to the inverted index, so that when the inverted index is queried according to the field characteristics corresponding to the key value, not only the data column corresponding to the storage information in the array corresponding to the key value can be queried, but also the data column that is a data column pair with the data column corresponding to the storage information can be obtained.

[0066] In the embodiment of the present application, by calculating the similarity based on the feature vectors of any two data columns, each data column is filtered to obtain a suspected matching pair whose similarity calculation result is greater than the similarity threshold, and the data in the suspected matching pair is extracted and verified to obtain the associated first data column and second data column, and the associated information is written into the array of the inverted index. A more refined index structure is constructed within the public data set, so that when the user queries based on the inverted index, a more refined search can be performed within the public data set, not only to query the data column corresponding to the information stored in the array corresponding to the key value, but also to obtain the data column associated with the data column corresponding to the stored information, which improves the query efficiency while also improving the comprehensiveness and accuracy of the query results.

[0067] See also Figure 4 , is a flow chart of an index construction method provided in Example 4 of the present application. Figure 4 As shown, in the above step S202, constructing an inverted index according to the field characteristics and storage information of each data column may include the following steps:

[0068] Step S401: If the public data set contains long text data, the long text data is compressed and summarized by using a trained language model to obtain a summarized document.

[0069] In an embodiment of the present application, long text data may refer to a data type containing a large amount of text content. For example, the long text data may be a paragraph containing multiple meanings. The trained language model may refer to a language model that has been trained for text compression and induction. For example, the trained language model may be a trained large language model (LargeLanguage Model, LLM). The summarized document may refer to data after compression and summarization of the long text data.

[0070] For example, a paragraph with multiple meanings can be compressed and summarized into one or two sentences of natural language as a summarized document by using a trained LLM model while maintaining less semantic loss.

[0071] Step S402: perform word segmentation processing on the field features of each data column to obtain a first word segmentation result of the corresponding data column.

[0072] Step S403, constructing an inverted index, writing the first word segmentation result of each data column into a blank key value of the inverted index, and writing the storage information of the corresponding data column into an array of the corresponding key value.

[0073] Step S404, obtaining storage information of each summarized document, performing word segmentation processing on each summarized document respectively, and obtaining a second word segmentation result corresponding to the summarized document.

[0074] Step S405, write the second word segmentation result of each summarized document into a blank key value of the inverted index, and write the storage information corresponding to the summarized document into an array corresponding to the key value.

[0075] In an embodiment of the present application, the first word segmentation result may refer to the individual words or phrases obtained by performing word segmentation processing on the field features, the storage information may refer to the storage location of the summarized document in the public data set, and the second word segmentation result may refer to the individual words or phrases obtained by performing word segmentation processing on the summarized document.

[0076] Specifically, first, the dictionary and inverted list in the inverted index structure can be constructed separately according to the index construction tool, and then the words or phrases obtained after the data column and the summarized document segmentation processing are written into the blank key values ​​corresponding to the dictionary, and a unique identifier is assigned to each word or phrase in the dictionary, and then the storage information corresponding to each word or phrase in the dictionary is written into the array of the key value corresponding to the inverted list.

[0077] In an embodiment of the present application, if a public data set contains long text data, the long text data is compressed and summarized to obtain a summarized document, and the field features of each data column and each summarized document are segmented to obtain a first segmentation result and a second segmentation result, and the first segmentation result and the second segmentation result are written into a blank key value of an inverted index, and the storage information corresponding to the first segmentation result and the second segmentation result is written into an array of key values ​​corresponding to the inverted index. By constructing an inverted index, it is possible to quickly locate the storage location of the corresponding word or phrase according to the inverted index item, which not only improves the efficiency of the query, but also because the inverted index is constructed based on the features of the data columns and long text data in the public data set, it is also possible to perform a more refined search within the public data set according to the inverted index, thereby improving the comprehensiveness and accuracy of the search results.

[0078] See also Figure 5 , is a flowchart of a field search method provided in Embodiment 5 of the present application, and the field search method is applied to Figure 1 The server in the embodiment 2 to the embodiment 4 is connected to the client to obtain the data to be queried sent by the client, and after the index construction method of any one of the above embodiments 2 to 4 is used to obtain the inverted index and the vector index. Figure 5 As shown, the field search method may include the following steps:

[0079] Step S501, obtaining the data to be queried, and determining the vector to be queried of the data to be queried.

[0080] In the embodiment of the present application, the data to be queried may refer to keyword data such as words or phrases to be queried, and the vector to be queried may refer to a vectorized representation of the data to be queried. Specifically, the data to be queried may be converted into a vector to be queried by a corresponding vector conversion model, such as a word embedding model and a Bert model.

[0081] Step S502: determine a first data set according to the data to be queried and the inverted index, and determine a second data set according to the vector to be queried and the vector index.

[0082] In the embodiment of the present application, the first data set may refer to a corresponding public data set obtained by querying an inverted index according to the data to be queried, and the second data set may refer to a corresponding public data set obtained by querying a vector index according to the vector to be queried.

[0083] Specifically, in the process of determining the first data set, first, prefix filtering is performed on the data to be queried to obtain the filtered data to be queried, then, the filtered data to be queried is used as an index item to query the inverted index to obtain the corresponding data columns and / or long text data, and finally, according to the data to be queried, the data columns and / or long text data obtained by the query are filtered to obtain the filtered data columns and / or long text data, and the public data set corresponding to the filtered data columns and / or long text data is determined to be the first data set.

[0084] In the process of determining the second data set, first, query parameters such as the query radius of the vector index and the number of query results returned can be set. Then, based on the set query parameters, the vector to be queried is used as the index item to query the vector index, and the similarity score between the vector to be queried and each feature vector in the vector index is calculated, and each feature vector is sorted according to the similarity score. Finally, based on the sorting result, the feature vectors with the same number as the query results are returned, and the public data set corresponding to each feature vector obtained by the query is determined to be the second data set.

[0085] Step S503: forming a candidate data set according to the first data set and the second data set.

[0086] Step S504: sort all the data in the candidate data set according to the data to be queried, and obtain the sorted data as the query result.

[0087] In the embodiment of the present application, the candidate data set may refer to a set of public data sets consisting of the first data set and the second data set, and the query result may refer to the query result obtained by performing a field search based on the data to be queried.

[0088] Specifically, first, the first data set obtained by querying the inverted index and the second data set obtained by querying the vector index are compared, duplicate public data sets are removed, and they are merged to obtain a candidate data set. Then, for any public data set in the candidate data sets, the correlation score between the data to be queried and the public data set is calculated. Finally, according to the correlation score, the public data sets in the candidate data sets are sorted, and the sorted result is the query result.

[0089] In the embodiment of the present application, a first data set is determined according to the data to be queried and the inverted index, a second data set is obtained according to the vector to be queried and the vector index corresponding to the data to be queried, a candidate data set is obtained according to the first data set and the second data set, and all data in the candidate data set are sorted according to the data to be queried, and the sorted data is obtained as the query result. Among them, by querying the inverted index and the vector index to obtain the candidate data set, a more refined search is performed within the public data set, the comprehensiveness and accuracy of the candidate data set obtained by the query are improved, and the candidate data set obtained by the query is finely sorted to obtain the query result, making the query result clearer and providing convenience for users to perform subsequent processing of the query results.

[0090] See also Figure 6 , is a flow chart of a field search method provided in Example 6 of the present application. Figure 6 As shown, in the above step S502, determining the first data set according to the data to be queried and the inverted index may include the following steps:

[0091] Step S601, prefix filtering is performed on the data to be queried to obtain the filtered data to be queried.

[0092] Step S602, obtaining candidate data according to the filtered data to be queried and the inverted index.

[0093] In an embodiment of the present application, prefix filtering may refer to a data processing technique for screening out data with a specific prefix during data query or processing, and candidate data may refer to data columns and / or long text data obtained by querying based on the filtered data to be queried and the inverted index.

[0094] Specifically, in the process of obtaining candidate data, first, string processing technology can be used to perform word segmentation on the query data to divide the query data into multiple fields, and then prefix filtering is performed on the query data based on the divided multiple fields to obtain filtered query data. Finally, the filtered query data is used as an index item to query the inverted index to obtain the corresponding data column, and / or the long text data is the candidate data.

[0095] For example, if the data to be queried is "unit name", the data to be queried is divided into multiple fields with a length of 1, and the divided data to be queried can be {"unit", "position", "name", "title"}. If the divided data to be queried is prefix filtered and the filtered data to be queried is "unit", then "unit" is used as the index item, the inverted index is queried, and the corresponding data column is obtained, and / or the long text data is the candidate data.

[0096] Step S603: filter the candidate data according to the data to be queried to obtain a first data set.

[0097] Specifically, in the process of obtaining the first data set, first, the data and long text data in the data column in the candidate data can be segmented by string processing technology to divide the candidate data into multiple candidate fields, and the data to be queried can be segmented to divide the data to be queried into multiple fields. For any candidate data, each candidate field in the candidate data is compared with multiple fields in the data to be queried by predefined operators, such as equal signs, comparison symbols, and inclusion relations, and the number of candidate fields in the candidate data that are the same as the fields of the data to be queried is recorded. Secondly, the recorded number is compared with the preset number. If the recorded number is If the number of candidate data exceeds the preset number, it is determined that the candidate data is similar to the data to be queried, and the data set where the candidate data is located is determined to be the candidate data set. Then, the candidate data whose number exceeds the preset number is determined for all records, and the term frequency-inverse document frequency value of each determined candidate data in all public data sets is calculated respectively. Finally, the term frequency-inverse document frequency value of each determined candidate data, the candidate data set corresponding to each determined candidate data, and the data to be queried are input into a pre-trained logistic regression model to obtain the confidence result of each candidate data set, and each candidate data set is sorted according to the confidence result to obtain the first data set by screening.

[0098] In the embodiment of the present application, prefix filtering is performed on the data to be queried to obtain the filtered data to be queried, candidate data is obtained based on the filtered data to be queried and the inverted index, and the candidate data is filtered based on the data to be queried to obtain a first data set. Among them, by prefix filtering the data to be queried and querying the inverted index based on the filtered data to be queried, the query range of the inverted index is narrowed, the efficiency of the query is improved, and the candidate data obtained by the query is filtered to obtain the first data set, which improves the accuracy of the public data set obtained by the query.

[0099] See also Figure 7 , is a flow chart of a field search method provided in Embodiment 7 of the present application. Figure 7 As shown, in the above step S504, all the data in the candidate data set are sorted according to the data to be queried, and the sorted data is obtained as the query result, which may include the following steps:

[0100] Step S701, performing word segmentation processing on the query data to obtain a third word segmentation result.

[0101] Step S702, calculating the relevance score between the third word segmentation result and each data in the candidate data set, and sorting all the data in the candidate data set according to the relevance score, and obtaining the sorted data as the query result.

[0102] In the embodiment of the present application, the third word segmentation result may refer to each word or phrase obtained by performing word segmentation processing on the query data.

[0103] Specifically, first, the query data is segmented to obtain each word or phrase as the third segmentation result. Secondly, for any word or phrase obtained from the segmentation of the query data and any candidate data set, the relevance score between the word or phrase and the candidate data set is calculated. Then, the relevance scores of all words or phrases obtained from the segmentation of the query data and the candidate data set are added together to obtain the relevance score of the candidate data set. Finally, the calculated relevance scores of the candidate data sets and the query data are sorted, and the sorted result is obtained as the query result.

[0104] In the embodiment of the present application, the query data is segmented to obtain a third segmentation result, the correlation score between the third segmentation result and each data in the candidate data set is calculated, and all the data in the candidate data set is sorted according to the correlation score, and the sorted data is obtained as the query result. Among them, the first data set obtained by the query inverted index and the second data set obtained by the query vector index are sorted by the query data, thereby improving the comprehensiveness and accuracy of the search results.

[0105] Corresponding to the index construction method of the above embodiment, Figure 8 The structure block diagram of the index construction device provided in the eighth embodiment of the present application is shown. The index construction device is applied to Figure 1 The server in the embodiment of the present invention is connected to the client to obtain the public data set sent by the client. For the convenience of explanation, only the part related to the embodiment of the present application is shown.

[0106] See also Figure 8 , the index building device comprises:

[0107] The feature extraction module 81 is used to obtain a public data set, and for any data column in the public data set, perform field extraction on the data column according to the field extraction method corresponding to the data type of the data column, obtain the field feature of the data column, and determine the feature vector corresponding to each field feature;

[0108] The index building module 82 is used to obtain the storage information of each data column in the public data set, build an inverted index according to the field characteristics and storage information of each data column, and build a vector index according to the feature vector and storage information of each data column.

[0109] Optionally, the index building device further includes:

[0110] A similarity calculation module is used to calculate the similarity of the feature vectors of any two data columns to obtain a similarity calculation result. If the similarity calculation result is greater than a similarity threshold, the two data columns are determined to be a suspected matching pair;

[0111] A matching module, configured to extract a preset number of first data from the first data column in the suspected matching pair, and match all the first data with the second data column in the suspected matching pair to obtain a matching result;

[0112] an association determination module, configured to count the proportion of the first data whose matching result is matched in the preset number, and if the proportion exceeds a proportion threshold, determine that the suspected matching pair has passed the verification, and form association information with the field features of the first data column and the second data column;

[0113] The association writing module is used to write the association information into the array of the first data column and the second data column in the inverted index after the first data column and the second data column are written into the inverted index.

[0114] Optionally, the index building device further includes:

[0115] An induction module, used for compressing and summarizing the long text data by using a trained language model to obtain a summarized document if the public data set contains long text data;

[0116] The first word segmentation module is used to perform word segmentation processing on the field features of each data column to obtain the first word segmentation result of the corresponding data column;

[0117] A first writing module is used to construct an inverted index, write the first word segmentation result of each data column into a blank key value of the inverted index, and write the storage information of the corresponding data column into an array of the corresponding key value;

[0118] The second word segmentation module is used to obtain the storage information of each summarized document, perform word segmentation processing on each summarized document, and obtain the second word segmentation result of the corresponding summarized document;

[0119] The second writing module is used to write the second word segmentation result of each summarized document into a blank key value of the inverted index, and write the storage information of the corresponding summarized document into the array corresponding to the key value.

[0120] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0121] Corresponding to the field search method in the above embodiment, Fig. 9 The structural block diagram of the field search device provided in the ninth embodiment of the present application is shown. The field search device is applied to Figure 1 The server in the embodiment of the present invention is connected to the client to obtain the data to be queried sent by the client. For the convenience of explanation, only the part related to the embodiment of the present application is shown.

[0122] See also Fig. 9 After the index construction method of any one of the above-mentioned embodiments 2 to 4 is used to obtain the inverted index and the vector index, the field search device includes:

[0123] A query acquisition module 91 is used to acquire the data to be queried and determine the query vector of the data to be queried;

[0124] A first determination module 92, configured to determine a first data set according to the to-be-queried data and the inverted index, and to determine a second data set according to the to-be-queried vector and the vector index;

[0125] A second determining module 93, configured to form a candidate data set according to the first data set and the second data set;

[0126] The result acquisition module 94 is used to sort all the data in the candidate data set according to the data to be queried, and obtain the sorted data as the query result.

[0127] Optionally, the first determining module 92 includes:

[0128] A prefix filtering unit, used for performing prefix filtering on the data to be queried to obtain filtered data to be queried;

[0129] An inverted query unit, used to obtain candidate data according to the filtered data to be queried and the inverted index;

[0130] The data filtering unit is used to filter the candidate data according to the data to be queried to obtain a first data set.

[0131] Optionally, the result acquisition module 94 includes:

[0132] A third word segmentation unit is used to perform word segmentation processing on the to-be-queried data to obtain a third word segmentation result;

[0133] The sorting unit is used to calculate the relevance score between the third word segmentation result and each data in the candidate data set, and sort all the data in the candidate data set according to the relevance score, and the sorted data is the query result.

[0134] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0135] Fig.10 This is a schematic diagram of the structure of a computer device provided in Example 10 of the present application. Fig.10 As shown, the computer device of this embodiment includes: at least one processor ( Fig.10 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned index building methods or the above-mentioned field search method embodiments are implemented.

[0136] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Fig.10 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0137] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0138] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of an operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments may also be an external storage device of a computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on a computer device. Further, the memory may also include both an internal storage unit of a computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as program codes of computer programs, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0139] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiment when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0140] The present application implements all or part of the processes in the above-mentioned embodiment method, and may also be completed through a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing the computer program product.

[0141] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0142] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0143] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0144] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0145] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An index construction method, characterized in that: The index building method comprises: Acquire a public data set, and for any data column in the public data set, perform field extraction on the data column according to a field extraction method corresponding to the data type of the data column, obtain field features of the data column, and determine a feature vector corresponding to each field feature; Obtain storage information of each data column in the public data set, build an inverted index according to the field characteristics and storage information of each data column, and build a vector index according to the feature vector and storage information of each data column; After obtaining the field features and feature vectors of the data column, the method further includes: Calculate the similarity of the feature vectors of any two data columns to obtain a similarity calculation result, and if the similarity calculation result is greater than a similarity threshold, determine that the two data columns are a suspected matching pair; Extracting a preset number of first data from the first data column in the suspected matching pair, matching all the first data with the second data column in the suspected matching pair, and obtaining a matching result; Counting the proportion of the first data whose matching result is matched in the preset number, if the proportion exceeds a proportion threshold, determining that the suspected matching pair has passed the verification, and forming association information with the field features of the first data column and the second data column; After the first data column and the second data column are written into the inverted index, writing the association information into an array of the first data column and the second data column in the inverted index; The index building method further includes: If the public data set contains long text data, the long text data is compressed and summarized by using the trained language model to obtain a summarized document; The step of constructing an inverted index according to the field characteristics and storage information of each data column includes: Perform word segmentation processing on the field features of each data column respectively to obtain the first word segmentation result of the corresponding data column; Construct an inverted index, write the first word segmentation result of each data column into a blank key value of the inverted index, and write the storage information of the corresponding data column into an array of the corresponding key value; Obtain storage information of each summarized document, perform word segmentation processing on each summarized document respectively, and obtain a second word segmentation result of the corresponding summarized document; The second word segmentation result of each summarized document is written into a blank key value of the inverted index, and the storage information of the corresponding summarized document is written into an array corresponding to the key value.

2. A field search method, characterized in that: After obtaining an inverted index and a vector index by using the index building method according to claim 1, the field search method includes: Acquire the data to be queried, and determine the vector to be queried of the data to be queried; Determine a first data set according to the data to be queried and the inverted index, and determine a second data set according to the vector to be queried and the vector index; forming a candidate data set according to the first data set and the second data set; According to the data to be queried, all the data in the candidate data set are sorted, and the sorted data obtained is the query result.

3. The field search method according to claim 2, characterized in that: The determining of the first data set according to the to-be-queried data and the inverted index includes: Performing prefix filtering on the data to be queried to obtain filtered data to be queried; Obtaining candidate data according to the filtered data to be queried and the inverted index; The candidate data are filtered according to the data to be queried to obtain a first data set.

4. The field search method according to claim 2, characterized in that: The step of sorting all the data in the candidate data set according to the data to be queried, and obtaining the sorted data as the query result, includes: Performing word segmentation processing on the data to be queried to obtain a third word segmentation result; The relevance score between the third word segmentation result and each data in the candidate data set is calculated, and all the data in the candidate data set are sorted according to the relevance score, and the sorted data obtained is the query result.

5. An index building device, characterized in that: The index building device comprises: A feature extraction module is used to obtain a public data set, and for any data column in the public data set, perform field extraction on the data column according to the field extraction method corresponding to the data type of the data column, obtain the field features of the data column, and determine the feature vector corresponding to each field feature; An index building module, used to obtain the storage information of each data column in the public data set, build an inverted index according to the field characteristics and storage information of each data column, and build a vector index according to the feature vector and storage information of each data column; A similarity calculation module is used to calculate the similarity of the feature vectors of any two data columns to obtain a similarity calculation result. If the similarity calculation result is greater than a similarity threshold, the two data columns are determined to be a suspected matching pair; A matching module, configured to extract a preset number of first data from the first data column in the suspected matching pair, and match all the first data with the second data column in the suspected matching pair to obtain a matching result; an association determination module, configured to count the proportion of the first data whose matching result is matched in the preset number, and if the proportion exceeds a proportion threshold, determine that the suspected matching pair has passed the verification, and form association information with the field features of the first data column and the second data column; an association writing module, configured to write the association information into an array of the first data column and the second data column in the inverted index after the first data column and the second data column are written into the inverted index; An induction module, used for compressing and summarizing the long text data by using a trained language model to obtain a summarized document if the public data set contains long text data; The first word segmentation module is used to perform word segmentation processing on the field features of each data column to obtain the first word segmentation result of the corresponding data column; A first writing module is used to construct an inverted index, write the first word segmentation result of each data column into a blank key value of the inverted index, and write the storage information of the corresponding data column into an array of the corresponding key value; The second word segmentation module is used to obtain the storage information of each summarized document, perform word segmentation processing on each summarized document, and obtain the second word segmentation result of the corresponding summarized document; The second writing module is used to write the second word segmentation result of each summarized document into a blank key value of the inverted index, and write the storage information of the corresponding summarized document into the array corresponding to the key value.

6. A field search device, characterized in that: After obtaining the inverted index and the vector index by adopting the index construction method according to claim 1, the field search device comprises: A query acquisition module, used to acquire the data to be queried and determine the vector to be queried of the data to be queried; A first determination module, configured to determine a first data set according to the data to be queried and the inverted index, and to determine a second data set according to the vector to be queried and the vector index; a second determining module, configured to form a candidate data set according to the first data set and the second data set; The result acquisition module is used to sort all the data in the candidate data set according to the data to be queried, and obtain the sorted data as the query result.

7. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the index building method according to claim 1 or the field search method according to any one of claims 2 to 4 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the index building method according to claim 1 or the field search method according to any one of claims 2 to 4 is implemented.

Images