Structured data retrieval method and system, electronic device and storage medium

Abstract

The application discloses a structured data retrieval method, a structured data retrieval system, electronic equipment and a storage medium, and relates to the technical field of computers.The method comprises the following steps: storing data in a set format, adopting a set word segmentation algorithm to segment a keyword to be retrieved, and obtaining similar words of each word in a word segmentation result; searching the data in the structured storage according to the keyword segmentation result and data attributes, merging the searched data, and sorting the merged data according to a set sorting mode. The application can effectively retrieve data.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a method for retrieving structured data, a system for retrieving structured data, a system, an electronic device, and a storage medium. Background Technology

[0002] Common retrieval methods can be broadly categorized into two types: one type is based on forward retrieval using structured data, which supports searching based on one or more attributes of the structured data and sorting the search results according to the specified attributes, but does not support inverted indexes and word segmentation search, and can only perform simple exact matching and fuzzy matching based on keywords; the other type supports inverted indexes and word segmentation search, but under normal circumstances, it can only search based on a single attribute of the structured data, and can only sort the search results based on a single attribute of the data.

[0003] It is evident that the above two common retrieval methods have the following problems:

[0004] (1) The two types of retrieval processing methods are mutually exclusive and cannot be integrated. Conventional retrieval methods cannot simultaneously take into account the unique and excellent features of both.

[0005] (2) When sorting the retrieved data, it is not possible to sort the data according to the next sorting attribute when the sorting weights of the previous attributes are similar.

[0006] (3) Existing search methods can only search based on the current keywords and cannot retrieve data of synonyms that match the keywords;

[0007] (4) Existing retrieval methods can only retrieve data that matches the keywords, and cannot intelligently retrieve data that matches the keywords' superordinate or subordinate terms. Summary of the Invention

[0008] In view of the deficiencies in the existing technology, the purpose of this invention is to provide a method for retrieving structured data, a system for retrieving structured data, a system, an electronic device, and a storage medium, which can realize effective data retrieval.

[0009] To achieve the above objectives, the present invention provides a method for retrieving structured data, specifically including the following steps:

[0010] The data is stored in a structured manner based on a set format, and a set word segmentation algorithm is used to segment the keywords to be searched, and similar words are obtained from each word in the segmentation results;

[0011] Based on the keyword segmentation results, the data is retrieved from the structured storage data according to data attributes, and the retrieved data is then merged.

[0012] According to the requirements of the set sorting method, the merged data is sorted using the corresponding sorting method.

[0013] Based on the above technical solutions,

[0014] The specified word segmentation algorithm is a word segmentation algorithm that integrates a statistical model-based word segmentation method and a deep learning-based end-to-end word segmentation method;

[0015] The word segmentation algorithm has a built-in dictionary containing multiple word groups, and the dictionary built-in by the word segmentation algorithm supports dynamic updates.

[0016] The similar words include synonyms and near-synonyms, as well as hypernyms or hyponyms.

[0017] Based on the above technical solution, the structured storage of data according to the set format specifically includes:

[0018] The attribute words and phrases in the data are segmented based on the set word segmentation algorithm to obtain the word segmentation results of the attribute words and phrases;

[0019] The metadata and attribute word segmentation results of the data are stored in JSON format.

[0020] Based on the above technical solution, the specific steps of retrieving data from structured storage according to data attributes based on keyword segmentation results and merging the retrieved data include:

[0021] Based on the similar words obtained when segmenting the keywords to be retrieved, determine the similarity relationship between the words in the keyword segmentation results;

[0022] Based on the similarity between words in the keyword segmentation results, the keyword segmentation results are grouped to obtain at least one set of search parameters;

[0023] Based on multi-threading technology, the search parameters within each group are retrieved from the structured storage data according to the BM25 algorithm to obtain all data that meets the requirements.

[0024] The retrieved data is merged and deduplicated to obtain preliminary search results.

[0025] Based on the above technical solution, the step of sorting the merged data according to the requirements of the set sorting method includes the following specific steps:

[0026] Based on the weight calculation algorithm, the weight values ​​of each attribute field that needs to be sorted in the preliminary search results data are calculated.

[0027] The calculated weight values ​​of the sorting attribute fields are assigned to the preliminary search result data, serving as the word weights of the sorting attribute in the preliminary search result data.

[0028] The weights of each ranking attribute in the preliminary retrieval results data are normalized using a normalization function formula.

[0029] Based on the requirements of the set sorting method, the weights of each sorting attribute in the preliminary search results data after normalization are scaled.

[0030] The weights of all ranking attributes and phrases in a single preliminary search result data after scaling are summed according to a set calculation function, and this sum is used as the final ranking weight of the current preliminary search result data.

[0031] Based on the final ranking weight of the preliminary search results data, the preliminary search results data are sorted to display the search results.

[0032] Based on the above technical solution, after normalizing the weights of each ranking attribute of the preliminary retrieval results data using a normalization function formula, the following is also included:

[0033] The weights of each ranking attribute of the preliminary retrieval results data are amplified based on the pre-trained exponential function calculation formula;

[0034] The weights of the amplified sorting attributes are converted into specified values ​​based on a preset piecewise function formula.

[0035] Based on the above technical solution, the scaling processing of the word and phrase weights of each ranking attribute in the normalized preliminary search result data according to the requirements of the set ranking method is as follows:

[0036] According to the requirements of the set sorting method, the weights of each sorting attribute term in the normalized preliminary search results data are multiplied by the corresponding weight coefficients to achieve scaling processing of the sorting attribute term weights.

[0037] This invention provides a structured data retrieval system, comprising:

[0038] The word segmentation module is used to store data in a structured manner based on a set format, and to segment the keywords to be searched using a set word segmentation algorithm, and to obtain similar words for each word in the segmentation results;

[0039] The retrieval module is used to retrieve data based on data attributes in structured storage data according to keyword segmentation results, merge the retrieved data, and perform deduplication during the merging process.

[0040] The sorting module is used to sort the merged data according to the set sorting method.

[0041] The present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the structured data retrieval method described above.

[0042] The present invention provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the structured data retrieval method described above.

[0043] Compared with the prior art, the advantages of the present invention are: it can perform retrieval based on one or more attributes of structured data, and supports exact match and fuzzy match retrieval, inverted index and word segmentation search. It can also support synonym retrieval and hypernym (hyponym) retrieval, and sort the retrieval results based on multiple attributes, so as to sort the results based on the next ranking attribute when the ranking weight of the previous attribute is similar. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of a structured data retrieval method according to an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of a structured data retrieval system according to an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0048] See Figure 1 As shown, this embodiment of the invention provides a method for retrieving structured data. By integrating multiple Chinese word segmentation algorithms, relevance scoring algorithms, weight function transformations, multithreading, and other technologies, it achieves data retrieval functionality and improves search performance. The retrieval method specifically includes the following steps:

[0049] S1: Structurally store the data based on a set format, segment the keyword to be retrieved using a set word segmentation algorithm, and obtain the similar words of each word in the segmentation result;

[0050] In the embodiment of the present invention, the set word segmentation algorithm is a word segmentation algorithm that combines a word segmentation method based on a statistical model and an end-to-end word segmentation method based on deep learning. The set word segmentation algorithm has a built-in dictionary containing multiple phrases, and the built-in dictionary of the set word segmentation algorithm supports dynamic update. That is, based on the full combination of the word segmentation method based on the statistical model and the end-to-end word segmentation method based on deep learning in advance, a proprietary word segmentation algorithm is constructed, and a dictionary containing a large number of phrases is built in, and at the same time, the built-in dictionary is allowed to be dynamically updated to update the word segmentation result in time. Similar words include synonyms, near synonyms, as well as hypernyms or hyponyms.

[0051] It should be noted that similar words refer to a group of words with the same meaning, which can be divided into two types: synonyms and near synonyms. Near synonyms refer to words with similar meanings but not exactly the same; while synonyms have exactly the same meaning. For example, the synonym of "potato" is "sweet potato", and "good-looking" and "beautiful" are near synonyms of each other. A hypernym refers to a topic word with a broader extension in concept. For example: "flower" is the hypernym of "fresh flower", and "plant" is the hypernym of "flower". A hyponym is a Chinese word meaning a topic word with a narrower intension in concept. For example: the hyponyms of "flower" include "rose", "lily", and the hypernym of "lily" is "flower".

[0052] In the present invention, structurally storing the data based on a set format specifically includes:

[0053] S101: Segment the attribute sentences of the data based on the set word segmentation algorithm to obtain the segmentation result of the attribute sentences;

[0054] S102: Store the metadata of the data and the segmentation result of the attribute sentences in the json structure (a lightweight data exchange format).

[0055] After structurally storing the metadata of the data and the segmentation result of the attribute sentences, the set word segmentation algorithm can be used to segment the input keyword to be retrieved, and the corresponding similar words can be obtained while segmenting. It should be noted that the set word segmentation algorithm is used both when storing the original data and when retrieving the stored data.

[0056] Regarding obtaining the similar words of each word in the segmentation result, the specific description is as follows: Suppose the keyword to be retrieved is "so beautiful and good-looking", then the words in the segmentation result after segmentation include "so", "beautiful", "and", "good-looking", and the similar words of "so", "beautiful", "and", and "good-looking" are obtained.

[0057] S2: Based on the keyword segmentation results, perform retrieval according to data attributes in the structured storage data, and merge the retrieved data; this step specifically involves:

[0058] S201: Based on the similar words obtained when segmenting the keywords to be retrieved, determine the similarity relationship between the words in the keyword segmentation results;

[0059] S202: Based on the similarity between words in the keyword segmentation results, group the keyword segmentation results to obtain at least one set of search parameters;

[0060] The following examples illustrate steps S201 and S202.

[0061] For example, in the search query "I like to eat pretty tomatoes", the synonyms for "pretty" are "beautiful" and "pretty", and the synonym for "tomato" is "tomato". Therefore, the following search combinations will be generated: 1: "I like to eat pretty tomatoes"; 2: "I like to eat beautiful tomatoes"; 3: "I like to eat pretty tomatoes"; 4: "I like to eat pretty tomatoes"; 5: "I like to eat beautiful tomatoes"; 6: "I like to eat pretty tomatoes". In total, there will be 6 different search combinations.

[0062] S203: Based on multi-threading technology, it retrieves all data that meets the requirements from the structured storage data by group, according to the BM25 algorithm (an algorithm used to evaluate the relevance between search terms and documents) for the search parameters within each group;

[0063] S204: Merge the retrieved data and perform deduplication to obtain preliminary search results.

[0064] S3: Sort the merged data according to the set sorting method. In actual use, there are multiple sorting rules, but only one sorting rule can be used at a time. After selecting the sorting rule, it will be automatically converted into the corresponding sorting logic, and then the data will be sorted according to that sorting logic. For example, you can choose sorting method one (sorting by data size), sorting method two (sorting by data creation time), or sorting method three (first sorting by data size, and then sorting by data creation time when the data sizes are similar).

[0065] In this invention, the merged data is sorted according to the requirements of the set sorting method, and the specific steps include:

[0066] S301: Based on the weight calculation algorithm, the weight values ​​of each attribute field that needs to participate in the ranking in the preliminary search results data are calculated. It should be noted that when generating search conditions, synonyms and near-synonyms have the same status. The difference lies in the ranking weight influence factors of synonyms and near-synonyms. Under the same conditions, the influence factor of synonyms is higher than that of near-synonyms.

[0067] S302: Assign the calculated weight values ​​of the sorting attribute fields to the preliminary search result data as the weights of the sorting attribute words in the preliminary search result data;

[0068] S303: Normalize the weights of each ranking attribute in the preliminary search results data using a normalization function formula;

[0069] S304: According to the requirements of the set sorting method, the weights of each sorting attribute in the preliminary search results data after normalization are scaled.

[0070] S305: The weights of all ranking attributes after scaling of a single preliminary search result data are summed according to a set calculation function to obtain the final ranking weight of the current preliminary search result data. For the summation method, the following example is given: For example, when calculating the final weight of data a (which has weight attribute values ​​“aα2, aβ2, aγ2”), the final weight a1 of data a can be approximated by the formula a1 = weight1 + (weight2)*aα2 + (weight3)*aβ2 + (weight4)*aγ2.

[0071] S306: Based on the final ranking weight of the preliminary search results data, sort the preliminary search results data to display the search results. For example, the corresponding data can be output to the user according to the input pagination parameters.

[0072] The following examples will be used to explain the process of steps S301 to S305 above in detail.

[0073] Suppose that dataset A has attribute fields α, β, and γ that need to be sorted. After filtering and deduplicating data from dataset A according to conditions, we obtain data a, b, and c. First, according to the rules, we obtain the initial sorting weights for the attribute fields α, β, and γ of data a, b, and c respectively, which are "aα, bα, cα", "aβ, bβ, cβ", and "aγ, bγ, cγ". Then, we normalize the data in each group according to the grouping "aα, bα, cα", "aβ, bβ, cβ", and "aγ, bγ, cγ", resulting in "aα1, bα1, cα1", "aβ1, bβ1, cβ1", and "aγ1, bγ1, cγ1". Next... For each group of data, the weight values ​​are amplified using a unique weight amplification function to obtain "aα2, bα2, α2", "aβ2, bβ2, cβ2", and "aγ2, bγ2, cγ2". Then, the weight values ​​are grouped again to obtain "aα2, aβ2, aγ2", "bα2, bβ2, bγ2", and "cα2, cβ2, cγ2". After grouping, the weight data in each group are calculated using a pre-trained calculation function, ultimately obtaining the final weight value a1 for data a, the final weight value b1 for data b, and the final weight value c1 for data c. Finally, data a, b, and c are sorted according to the values ​​of a1, b1, and c1.

[0074] In this embodiment of the invention, after normalizing the weights of each ranking attribute of the preliminary retrieval result data using a normalization function formula, the method further includes:

[0075] S3011: The weights of each ranking attribute word and phrase in the preliminary retrieval result data are amplified based on the pre-trained exponential function calculation formula. It should be noted that in this invention, the weights of each ranking attribute word and phrase in the preliminary retrieval result data are first normalized using a normalization function formula because the original ranking weights vary too much, and if normalization is not performed, the ranking results will be distorted.

[0076] S3012: Based on the preset piecewise function formula, the amplified ranking attribute word weights are converted into specified values.

[0077] It should be noted that piecewise functions may exist in the amplification process (that is, the process of transforming the data "aα1, bα1, cα1", "aβ1, bβ1, cβ1", "aγ1, bγ1, cγ1" into the numbers "aα2, bα2, cα2", "aβ2, bβ2, cβ2", "aγ2, bγ2, cγ2"), but not only piecewise functions exist.

[0078] In order to enhance the data's discernibility and ranking accuracy, the weights of each ranking attribute in the preliminary search results are first normalized using a normalization function formula. Then, an optimized and trained exponential function is used to amplify these weights. Finally, the amplified weights are converted into specified values ​​using a dynamically set piecewise function formula. Each weight is converted into a specified value, and these converted values ​​may be the same or different.

[0079] In this embodiment of the invention, according to the requirements of the set sorting method, the weights of each sorting attribute words and phrases in the normalized preliminary search result data are scaled. Specifically, according to the requirements of the set sorting method, the weights of each sorting attribute words and phrases in the normalized preliminary search result data are multiplied by the corresponding weight coefficients to achieve the scaling of the weights of the sorting attribute words and phrases.

[0080] This invention uses an improved custom word segmentation algorithm that integrates multiple word segmentation methods to segment and retrieve data information. Then, it calculates the final ranking weight of the filtered data through multiple function transformations and combinations, thereby expanding the ranking and recognition of the data and ultimately achieving the retrieval of the data retrieval target.

[0081] The structured data retrieval method of this invention can perform retrieval based on one or more attributes of the structured data. It supports exact match and fuzzy match retrieval, inverted index and word segmentation search, and can also support synonym retrieval and hypernym (hyponym) retrieval. It can also sort the retrieval results based on multiple attributes. For example, if data A is sorted based on attributes a, b, and c, it can be sorted based on the weight of attribute b if the weight of attribute a is similar, and so on, if the weight of attribute b is still similar, then it can be sorted based on attribute c, and so on.

[0082] See Figure 2 As shown in the figure, an embodiment of the present invention provides a structured data retrieval system, including a word segmentation module, a retrieval module, and a sorting module.

[0083] The word segmentation module stores data in a structured manner based on a set format, uses a set word segmentation algorithm to segment the keywords to be retrieved, and obtains similar words for each word in the segmentation results. The retrieval module searches the structured data based on data attributes according to the keyword segmentation results, merges the retrieved data, and performs deduplication during the merging process. The sorting module sorts the merged data according to the set sorting method. It should be noted that the word segmentation module is used in both the data storage and retrieval processes, and the sorting module also refers to the keyword segmentation results.

[0084] In one possible implementation, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the structured data retrieval method described above.

[0085] In one possible implementation, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the structured data retrieval method described above.

[0086] In this embodiment of the invention, the word segmentation algorithm is set to be a word segmentation algorithm that integrates a word segmentation method based on a statistical model and an end-to-end word segmentation method based on deep learning; the word segmentation algorithm is set to have a built-in dictionary containing multiple word groups, and the dictionary built-in by the word segmentation algorithm supports dynamic updates.

[0087] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

[0088] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

Claims

1. A method for retrieving structured data, characterized in that, Specifically, the following steps are included: The data is stored in a structured manner based on a set format, and a set word segmentation algorithm is used to segment the keywords to be searched, and similar words are obtained from each word in the segmentation results; Based on the keyword segmentation results, the data is retrieved from the structured storage data according to data attributes, and the retrieved data is then merged. According to the requirements of the set sorting method, the merged data is sorted using the corresponding sorting method; The steps of retrieving data based on data attributes from structured storage according to keyword segmentation results and merging the retrieved data include: Based on the similar words obtained when segmenting the keywords to be retrieved, determine the similarity relationship between the words in the keyword segmentation results; Based on the similarity between words in the keyword segmentation results, the keyword segmentation results are grouped to obtain at least one set of search parameters; Based on multi-threading technology, the search parameters within each group are retrieved from the structured storage data according to the BM25 algorithm to obtain all data that meets the requirements. The retrieved data is merged and deduplicated to obtain preliminary search results. The step of sorting the merged data according to the set sorting method requirements includes the following specific steps: Based on the weight calculation algorithm, the weight values ​​of each attribute field that needs to be sorted in the preliminary search results data are calculated. The calculated weight values ​​of the sorting attribute fields are assigned to the preliminary search result data, serving as the word weights of the sorting attribute in the preliminary search result data. The weights of each ranking attribute in the preliminary retrieval results data are normalized using a normalization function formula. Based on the requirements of the set sorting method, the weights of each sorting attribute in the preliminary search results data after normalization are scaled. The weights of all ranking attributes and phrases in a single preliminary search result data after scaling are summed according to a set calculation function, and this sum is used as the final ranking weight of the current preliminary search result data. Based on the final ranking weight of the preliminary search results data, the preliminary search results data are sorted to display the search results.

2. The method for retrieving structured data as described in claim 1, characterized in that: The specified word segmentation algorithm is a word segmentation algorithm that integrates a statistical model-based word segmentation method and a deep learning-based end-to-end word segmentation method; The word segmentation algorithm has a built-in dictionary containing multiple word groups, and the dictionary built-in by the word segmentation algorithm supports dynamic updates. The similar words include synonyms and near-synonyms, as well as hypernyms or hyponyms.

3. The method for retrieving structured data as described in claim 1, characterized in that, The structured storage of data based on a defined format specifically includes: The attribute words and phrases in the data are segmented based on the set word segmentation algorithm to obtain the word segmentation results of the attribute words and phrases; The metadata and attribute word segmentation results of the data are stored in JSON format.

4. The method for retrieving structured data as described in claim 1, characterized in that, After normalizing the weights of each ranking attribute in the preliminary retrieval results data using a normalization function formula, the process also includes: The weights of each ranking attribute of the preliminary retrieval results data are amplified based on the pre-trained exponential function calculation formula; The weights of the amplified sorting attributes are converted into specified values ​​based on a preset piecewise function formula.

5. The method for retrieving structured data as described in claim 1, characterized in that, The scaling process for the word and phrase weights of the normalized preliminary search results data, based on the requirements of the set sorting method, is as follows: According to the requirements of the set sorting method, the weights of each sorting attribute term in the normalized preliminary search results data are multiplied by the corresponding weight coefficients to achieve scaling processing of the sorting attribute term weights.

6. A structured data retrieval system, characterized in that, include: The word segmentation module is used to store data in a structured manner based on a set format, and to segment the keywords to be searched using a set word segmentation algorithm, and to obtain similar words for each word in the segmentation results; The retrieval module is used to retrieve data based on data attributes in structured storage data according to keyword segmentation results, merge the retrieved data, and perform deduplication during the merging process. The sorting module is used to sort the merged data according to the set sorting method requirements and the corresponding sorting method. The steps of retrieving data based on data attributes from structured storage according to keyword segmentation results and merging the retrieved data include: Based on the similar words obtained when segmenting the keywords to be retrieved, determine the similarity relationship between the words in the keyword segmentation results; Based on the similarity between words in the keyword segmentation results, the keyword segmentation results are grouped to obtain at least one set of search parameters; Based on multi-threading technology, the search parameters within each group are retrieved from the structured storage data according to the BM25 algorithm to obtain all data that meets the requirements. The retrieved data is merged and deduplicated to obtain preliminary search results. The step of sorting the merged data according to the set sorting method requirements includes the following specific steps: Based on the weight calculation algorithm, the weight values ​​of each attribute field that needs to be sorted in the preliminary search results data are calculated. The calculated weight values ​​of the sorting attribute fields are assigned to the preliminary search result data, serving as the word weights of the sorting attribute in the preliminary search result data. The weights of each ranking attribute in the preliminary retrieval results data are normalized using a normalization function formula. Based on the requirements of the set sorting method, the weights of each sorting attribute in the preliminary search results data after normalization are scaled. The weights of all ranking attributes and phrases in a single preliminary search result data after scaling are summed according to a set calculation function, and this sum is used as the final ranking weight of the current preliminary search result data. Based on the final ranking weight of the preliminary search results data, the preliminary search results data are sorted to display the search results.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the structured data retrieval method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the structured data retrieval method as described in any one of claims 1 to 5.

Images