Data query method and device and computer equipment

Abstract

The invention discloses a data query method and device and computer equipment. The method comprises the steps that a query statement is received, and the query statement is used for querying target data from multiple data texts stored in a cloud; a target index table of a plurality of category text sets in the cloud storage is obtained, the target index table at least comprises a text set core word of each category text set, the text set core words are used for representing subject information of the category text sets, and the plurality of category text sets are divided according to text contents of the plurality of data texts and keywords of the plurality of data texts; and obtaining the similarity between the query statement and the text set core word of each category text set in the target index table, and querying target data from the plurality of data texts according to the similarity. According to the method and the device, the technical problem of relatively low information retrieval efficiency in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of information retrieval technology, and in particular to a data query method, device and computer equipment. Background Art

[0002] With the popularization of the Internet and the improvement of the degree of informatization, the amount of data generated by the Internet per second is growing exponentially. Traditional storage and retrieval methods cannot meet the storage and retrieval query requirements of massive data. The emergence of cloud storage technology and distributed computing and storage technology has effectively solved the problem of explosive growth of data. However, with the development of the information age, users' demand for data is also constantly changing. Users hope to quickly find and obtain the required information from cloud storage data. Existing retrieval methods such as B+ tree index, hash index and full-text index, etc., mostly extract keywords or keywords from files, and then form corresponding index tables in a corresponding manner. Then, when searching, the keywords / words associated with the required query content are matched in the index table, and finally the corresponding files are fed back through the corresponding method of the index table. However, there are also various shortcomings. For example, B+ tree index may cause frequent splitting and reorganization of the tree for the current rapid data update, which will cause its performance to fluctuate; hash index also has certain difficulties for today's massive data, because it is difficult to design a general hash function suitable for diverse data; similarly, full-text index is also difficult to achieve fast query. Summary of the invention

[0003] The embodiments of the present application provide a data query method, apparatus and computer device to at least solve the technical problem of low information retrieval efficiency in the related art.

[0004] According to one aspect of an embodiment of the present application, a data query method is provided, comprising: receiving a query statement, the query statement being used to query target data from multiple data texts stored in a cloud; obtaining a target index table of multiple category text sets in the cloud storage, the target index table containing at least a corpus core word of each category text set, the corpus core word being used to characterize the main information of the category text set, the multiple category text sets being divided according to the text content of the multiple data texts and keywords of the multiple data texts; obtaining a similarity between the query statement and the corpus core word of each category text set in the target index table, and querying the target data from the multiple data texts according to the similarity.

[0005] Optionally, obtaining a target index table of multiple category text sets in the cloud storage includes: constructing an index entry for each data text based on the word frequency-inverse document frequency corresponding to the keywords of the multiple data texts, the file name of each data text, the creation time of each data text and the custom tag of each data text, wherein the custom tag is used to characterize the description information of the data text; performing word segmentation processing on each data text to obtain multiple phrases corresponding to each data text; mapping the multiple phrases corresponding to each data text to multiple word vectors, and determining the content similarity between each data text in the multiple data texts based on the multiple word vectors corresponding to each data text; determining an integration possibility index between each data text in the multiple data texts based on the content similarity between each data text in the multiple data texts and the similarity between the index entries of each data text; combining each data text in the multiple data texts in pairs based on the integration possibility index between each data text to obtain multiple groups of data texts; combining the multiple groups of data texts into multiple category text sets; and generating the target index table based on the multiple category text sets.

[0006] Optionally, the method further includes: obtaining the similarity between the file names of each data text; obtaining the similarity between the custom tags of each data text; and determining the similarity between the index entries of each data text based on the similarity between the custom tags and the similarity between the file names of each data text.

[0007] Optionally, obtaining the similarity between the file names of each data text includes: segmenting the file names of each data text to obtain multiple phrases corresponding to the file names, and removing stop words in the multiple phrases corresponding to the file names to obtain valid words corresponding to the file names; summing the word vectors of the valid words corresponding to the file names to obtain the sentence vectors corresponding to the file names of each data text; and determining the similarity between the file names of each data text based on the sentence vectors corresponding to the file names of each data text.

[0008] Optionally, obtaining the similarity between the custom labels of each data text includes: retrieving the custom label of each data text, and calculating the cosine similarity between the custom labels of each data text respectively; constructing a similarity matrix of the custom labels between each pair of data texts in the multiple data texts according to the cosine similarity between the custom labels of each data text, wherein the similarity matrix is ​​used to characterize the cosine similarity between the multiple custom labels corresponding to the two data texts; extracting the maximum value of each column and the value of the column corresponding to the maximum value from the similarity matrix; and determining the cosine similarity between the multiple custom labels corresponding to the two data texts based on the maximum value of each column and the value of the column corresponding to the maximum value.

[0009] Optionally, the multiple groups of data texts are combined into multiple category text sets, including: combining two data texts whose integration possibility index is higher than a preset threshold into a data text combination; obtaining the integration possibility index of each pair of data text combinations in the multiple groups of data text combinations; and combining the multiple groups of data text combinations according to the integration possibility index of each pair of data text combinations to obtain the multiple category text sets.

[0010] Optionally, the method also includes: obtaining index terms for each data text in each category text set; determining the description text of each category text set based on the file name, keywords and custom tags in the index terms of each data text; obtaining the word frequency-inverse document frequency of each phrase in the description text of each category text set, and determining the phrase in the description text whose word frequency-inverse document frequency is greater than the preset word frequency-inverse document frequency as the core word of the collection.

[0011] Optionally, the target index table is generated based on the multiple category text sets, including: obtaining the creation time of each data text, determining the collection period of each category text set according to the creation time, and determining the time prefix of each category text set according to the collection period; sorting the index entries of each data text in each category text set according to the number of strokes of the file name of each data text in the each category text set to obtain the retrieval appendix of each category text set; combining the description text, the time prefix and the retrieval appendix to obtain the index document of each category text set; and determining the target index table based on the index document of each category text set.

[0012] According to another aspect of an embodiment of the present application, a data query device is also provided, including: a receiving module, used to receive a query statement, wherein the query statement is used to query target data from multiple data texts stored in the cloud; an acquisition module, used to obtain a target index table of multiple category text sets in the cloud storage, wherein the target index table contains at least a corpus core word of each category text set, wherein the corpus core word is used to characterize the main information of the category text set, and the multiple category text sets are divided according to the text content of the multiple data texts and the keywords of the multiple data texts; a query module, used to obtain the similarity between the query statement and the corpus core word of each category text set in the target index table, and query the target data from the multiple data texts according to the similarity.

[0013] According to another aspect of the embodiment of the present application, a computer device is provided, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor is connected to the memory and is used to execute the above-mentioned data query method.

[0014] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned data query method by running the computer program.

[0015] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above data query method when executed by a processor.

[0016] In an embodiment of the present application, a query statement is received, the query statement is used to query target data from multiple data texts stored in the cloud; a target index table of multiple category text sets in the cloud storage is obtained, the target index table at least contains the core words of the corpus of each category text set, the core words of the corpus are used to characterize the main information of the category text set, and the multiple category text sets are divided according to the text content of the multiple data texts and the keywords of the multiple data texts; the similarity between the query statement and the core words of the corpus of each category text set in the target index table is obtained, and the target data is queried from the multiple data texts according to the similarity, so as to achieve the purpose of dividing semantically related or similar data texts into one index entry according to the category text set, thereby achieving the purpose of accurate and efficient information retrieval, thereby achieving the technical effect of improving information retrieval efficiency, and thus solving the technical problem of low information retrieval efficiency in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 is a hardware structure block diagram of a computer terminal for implementing a data query method according to an embodiment of the present application;

[0019] Figure 2 is a flow chart of a data query method according to an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of index terms of a data text according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of a similarity matrix according to an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of multiple parameters in a tag similarity calculation formula according to an embodiment of the present application;

[0023] Figure 6 is a schematic diagram of an index document according to an embodiment of the present application;

[0024] Figure 7 is a target index representation according to an embodiment of the present application;

[0025] Figure 8 is a flow chart of another data query method according to an embodiment of the present application;

[0026] Fig. 9 It is a structural diagram of a data query device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] The information collected in the embodiments of the present application is information and data authorized by the account or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for the account to choose to authorize or reject the automated decision-making results; if the account chooses to reject, it enters the expert decision-making process.

[0030] In order to solve the problems existing in the related art, the present application embodiment provides a data query method, which can be run on Figure 1 In the computer terminal shown, the computer terminal is explained below.

[0031] The data query method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a data query method. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1Different configurations are shown.

[0032] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0033] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data query method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned data query method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0034] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0035] The display may be, for example, a touch screen liquid crystal display (LCD), which may enable the account to interact with the account interface of the computer terminal 10 .

[0036] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0037] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a data query method. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] Figure 2 is a flow chart of a data query method according to an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0039] Step S202, receiving a query statement, wherein the query statement is used to search for target data from a plurality of data texts stored in the cloud;

[0040] In step S202, multiple data texts need to be pre-processed before being uploaded to the cloud storage.

[0041] The specific preprocessing process is as follows: Collect multiple data texts, each of which contains but is not limited to file name, creation time, file content, user-defined tags and other related information. Extract keywords from all file contents and file names, and obtain the corresponding term frequency-inverse document frequency (TF-IDF value).

[0042] The file content and file name are segmented to obtain numerous phrases; the TF-IDF value of each phrase is calculated; then, the phrases above the preset threshold are selected as quasi-keywords; by comparing with the words in the preset stop word list (such as the Chinese stop word list, the Harbin Institute of Technology stop word list, etc.), the stop words in the quasi-keywords are removed, and finally the keywords of the data text and the corresponding TF-IDF values ​​are obtained.

[0043] Step S204, obtaining a target index table of multiple category text sets in the cloud storage, wherein the target index table at least contains a core word of each category text set, wherein the core word of the text set is used to characterize the subject information of the category text set, and the multiple category text sets are divided according to the text content of the multiple data texts and the keywords of the multiple data texts;

[0044] It should be noted that user-defined tags refer to descriptive tags or keywords that users manually add when uploading or creating files for the convenience of management and retrieval. These tags can reflect the subject, content, purpose, project ownership, and other information of the file. For example, for a document about yoga practice, users may add tags such as "yoga", "fitness", and "leisure". In this way, in subsequent searches, even if the specific title or content of the file is not remembered, these tags can be used to quickly locate the file. The TF-IDF value can be determined based on the word frequency value and the inverse text frequency index, for example: TF*IDF. An index table is a separate, physical storage structure that sorts the values ​​of one or more columns in a database table; the role of an index is equivalent to the catalog of a book, and the required content can be quickly found according to the page number in the catalog. An index table is a way of storing index data according to index attributes, which can improve query performance. It can be understood that a target index table is an index table.

[0045] Step S206, obtaining the similarity between the query statement and the core words of the text set of each category in the target index table, and querying the target data from the multiple data texts according to the similarity.

[0046] In the related technology, the keywords of the text and the related descriptive information are directly integrated to form an index table. If a simple integration method is used, when faced with massive data, the index table is relatively large and the retrieval workload is large. If a complex integration method such as a B+ tree is used, when a large number of data texts are added or modified, the tree needs to be rebuilt. With today's massive data fluctuations, its performance will fluctuate frequently.

[0047] The present application achieves the purpose of accurately and efficiently performing information retrieval by dividing semantically related or similar data texts into one index entry through the above steps S202 to S206, thereby achieving the technical effect of improving information retrieval efficiency and solving the technical problem of low information retrieval efficiency in related technologies. The following is a detailed description.

[0048] This application integrates similar texts through their keywords and description information (collectively referred to as index terms) to establish a high-level index entry for them, and then organizes the index entries of many different categories of texts (with low similarity) to construct a high-level index table (target index table).

[0049] When adding data text, you can directly add it to the index terms below this entry based on the similarity or correlation between its keywords and description information and the index entries in the target index table; similarly, when searching, first find similar or related index entries (index of category text set) in the target index table, and then find the most similar and related entries among the many index terms below it. Cloud storage returns the query data without leaking the privacy of text information.

[0050] In step S204 of the above data query method, obtaining the target index table of multiple category text sets in the cloud storage includes: constructing the index term of each data text according to the word frequency-inverse document frequency corresponding to the keywords of the multiple data texts, the file name of each data text, the creation time of each data text and the custom tag of each data text, wherein the custom tag is used to characterize the description information of the data text; performing word segmentation processing on each data text to obtain multiple phrases corresponding to each data text; mapping the multiple phrases corresponding to each data text to multiple word vectors, and determining the content similarity between each data text in the multiple data texts according to the multiple word vectors corresponding to each data text; determining the integration possibility index between each data text in the multiple data texts based on the content similarity between each data text in the multiple data texts and the similarity between the index terms of each data text; combining each data text in the multiple data texts in pairs based on the integration possibility index between each data text to obtain multiple groups of data texts; combining the multiple groups of data texts into multiple category text sets; and generating the target index table based on the multiple category text sets.

[0051] In cloud storage, there are multiple data texts, among which there are many related data texts, or similar texts describing the same thing. Through the keywords and description information of related data texts and similar texts (collectively referred to as index terms), they are integrated with the content of each data text to form an index document, and a high-level index entry is built for them, and all texts are organized into a category text set (one text set corresponds to one index entry). After that, the index entries of many different category text sets are sorted to build the target index table.

[0052] Specifically, first, retrieve the keywords of all data texts and the corresponding TF-IDF values, file names, creation times, and user-defined tags and other related information to form index entries corresponding to each data text. Figure 3 A schematic diagram of index terms of a data text is shown, such as Figure 3As shown, the index terms include: the file address of the data text, the file name of the data text, the file creation time, the keywords of the data text and their corresponding TF-IDF values, and the custom tags of the data text.

[0053] Taking data text α and data text β as an example, the content similarity R between data text α and data text β is α,β It can be determined by the following formula:

[0054]

[0055] Where N α+β Represents the total number of word vectors in data text α and data text β, Represents the word vector of phrase i in the data text α, Represents the word vector of phrase j in text β; Represents the calculation of word vector With word vector The cosine similarity of .

[0056] It can be understood that word vectors are obtained by mapping phrases to real vectors in the semantic space. Phrases with larger cosine similarity have more similar semantics, that is, the two phrases are synonyms or even synonyms. The more similar or identical phrases there are in the two data texts, the higher the similarity between the two texts. Furthermore, the higher the proportion of phrases that appear in both data texts in the two texts, the higher the similarity or correlation between the two texts.

[0057] In some embodiments of the present application, the similarity between index terms of data texts can be determined in the following ways, for example: obtaining the similarity between the file names of each data text; obtaining the similarity between the custom tags of each data text; and determining the similarity between the index terms of each data text based on the similarity between the custom tags and the similarity between the file names of each data text.

[0058] Among them, the specific steps for determining the similarity between the file names of each data text are as follows: segment the file name of each data text to obtain multiple phrases corresponding to the file name, and remove stop words in the multiple phrases corresponding to the file name to obtain valid words corresponding to the file name; sum up the word vectors of the valid words corresponding to the file name to obtain the sentence vector corresponding to the file name of each data text; determine the similarity between the file names of each data text according to the sentence vector corresponding to the file name of each data text.

[0059] Specifically, we retrieve the valid phrases (all phrases except stop words) from the file names of data text α and data text β respectively, and calculate the vector sum of the word vectors corresponding to the phrases to obtain the sentence vectors corresponding to the file names of the two texts. The smaller the angle difference between the two sentence vectors in the semantic space, the closer the semantics of the two are, indicating that the file names of the two texts are more similar. Based on this, we calculate the similarity between the file names of data text α and data text β. α,β , the specific calculation method is shown in the following formula:

[0060]

[0061] In the formula, The word vector of phrase a in the file name of text α; a α represents the number of valid phrases in the file name of text α; The word vector of phrase b in the file name of text β; b β Represents the number of valid phrases in the file name of text β. The +1 is used to adjust the cosine similarity to avoid non-positive values ​​that lead to inaccurate judgment.

[0062] In addition, the similarity between the custom labels of each data text can be determined in the following manner: calling the custom labels of each data text, and calculating the cosine similarity between the custom labels of each data text respectively; constructing a similarity matrix of the custom labels between each pair of data texts in the multiple data texts according to the cosine similarity between the custom labels of each data text, and the similarity matrix is ​​used to characterize the cosine similarity between the multiple custom labels corresponding to the two data texts; extracting the maximum value of each column and the value of the column corresponding to the maximum value from the similarity matrix; determining the cosine similarity between the multiple custom labels corresponding to the two data texts based on the maximum value of each column and the value of the column corresponding to the maximum value.

[0063] Taking the similarity between the custom labels of data text α and data text β as an example, the custom labels of data text α and data text β are retrieved, and then the cosine similarity of the word vectors of their many custom labels is calculated respectively, and their similarity matrix is ​​constructed. Figure 4 The similarity matrix of the custom labels of a data text α and a data text β is shown as Figure 4 As shown, the first row represents multiple custom labels of data text α: label 1, label 2, label 3 and label 4, and the first column represents multiple custom labels of data text β: label a, label b, label c and label d.

[0064] Filter the labels of data text β that are most similar to the many custom labels of data text α and the corresponding cosine similarity, that is, the maximum value of each column in the similarity matrix and the label of the text β in the corresponding row. l,max represents the maximum value in column l, H l,max Indicates the row corresponding to the maximum value in column l.

[0065] Then select the labels of text α that are most similar to the many custom labels of text β and the corresponding cosine similarity, that is, the maximum value of each row in the similarity matrix and the label of text α in the corresponding column. h,max represents the maximum value in row h, L h,max Indicates the column corresponding to the maximum value in row h.

[0066] If text α is similar to text β, the similarity between their labels should also be relatively high. For example, if the label with the highest similarity to label 1 in the matrix is ​​label a, then the label most similar to label a should also be label 1. That is, in the similarity matrix, the maximum value of each row should also be the maximum value of the corresponding column. Based on this, the label similarity K of the custom labels of text α and text β is calculated. α,β :

[0067]

[0068] Where n α,l Represents the number of custom labels of text α, that is, the number of columns of the similarity matrix; s l,max represents the maximum value in column l, H l,max Indicates the row corresponding to the maximum value in column l; s h,max represents the maximum value in row h (h = H l,max ), L h,max Indicates the column corresponding to the maximum value in row h (h = H l,max ). AND(a,b) represents a logical operation on a and b. If they are the same, the output is 1, and if they are different, the output is 0. The +1 is used to adjust the cosine similarity to avoid inaccurate judgment caused by non-positive values.

[0069] Figure 5 A schematic diagram showing multiple parameters in the label similarity calculation formula.

[0070] In an optional manner, the similarity C between the index terms of the data text α and the data text β is α,β It can be determined by the following formula:

[0071] C α,β =name α,β +K α,β +μ[exp(-|TI g,α -TI g,β |)×2]

[0072] In the formula, name α,β K represents the name similarity between text α and text β in file names, which is used to describe the cosine similarity of the sentence vectors composed of the two file names; α,β Represents the label similarity of the custom label of text α and text β; TI g,α represents the TF-IDF value of keyword g in text α; TI g,β represents the TF-IDF value of keyword g in text β; μ[F g ] represents the F corresponding to many g g Calculate the average. The ×2 is to unify the similarity of keywords with the similarity of file names and custom tags to [0,2].

[0073] It should be noted that the more similar the file names of two data texts are, the more likely their contents are similar; secondly, the more similar the custom tags of the two texts are, the more similar people's perceptions of the two data texts are; thirdly, the closer the TF-IDF values ​​corresponding to the same keyword in the two data texts are, the more similar the two texts are in describing important content.

[0074] Optionally, the multiple groups of data texts are combined into multiple category text sets, including: combining two data texts whose integration possibility index is higher than a preset threshold into a data text combination; obtaining the integration possibility index of each pair of data text combinations in the multiple groups of data text combinations; and combining the multiple groups of data text combinations according to the integration possibility index of each pair of data text combinations to obtain the multiple category text sets.

[0075] Specifically, the more similar the two texts are in content, and the higher the correlation between them in the index, the higher the correlation between the two texts, and the higher the possibility that they can be searched at the same time, so they can be summarized and described by the same phrase at a higher level. Based on this, calculate the integration possibility Z of text α and text β α,β :

[0076] Z α,β =sigmoid(R α,β ×C α,β )

[0077] In the formula, R α,β represents the similarity between text α and text β in content; C α,β Indicates the correlation between text α and text β in the index.

[0078] Threshold Y Z Screening the integration possibility Z of text α and text β α,β , default Y Z=0.65. If the integration probability is greater than or equal to the threshold Y Z , then the text α and the text β are marked and recorded as a group of similar texts. If the integration probability is less than the threshold Y Z , then it is considered that text α is not similar to text β.

[0079] Traverse all the texts to obtain many similar texts that are similar to each other (two data texts in the data text combination are similar texts to each other). If there are repeated texts in two or more groups of similar texts, it is considered that these texts are similar or related. Multiple similar or related texts are integrated to form a category text set. Finally, multiple category text sets are obtained.

[0080] It should be noted that to determine whether there is repeated text in the data text combination, the above-mentioned method of calculating the data text similarity can be used, or the content similarity of the data text can be directly compared. When the content similarity is higher than the content similarity threshold, it can also be determined that there is repeated text in the data text.

[0081] Optionally, the method also includes: obtaining index terms for each data text in each category text set; determining the description text of each category text set based on the file name, keywords and custom tags in the index terms of each data text; obtaining the word frequency-inverse document frequency of each phrase in the description text of each category text set, and determining the phrase in the description text whose word frequency-inverse document frequency is greater than the preset word frequency-inverse document frequency as the core word of the collection.

[0082] The specific process of generating the target index table based on the multiple category text sets is as follows: obtaining the creation time of each data text, determining the collection period of each category text set according to the creation time, and determining the time preamble of each category text set according to the collection period; sorting the index entries of each data text according to the number of strokes of the file name of each data text in each category text set to obtain the retrieval appendix of each category text set; combining the description text, the time preamble and the retrieval appendix to obtain the index document of each category text set; determining the target index table based on the index document of each category text set.

[0083] Taking a category text set as an example, the process of obtaining its index entries is as follows:

[0084] Retrieve the index entries of each text in the category text set, record all the file names, keywords and custom tags, and form the description text of this category text set. Calculate the TF-IDF value of each phrase in the text, and then mark the phrases greater than or equal to the threshold (preset as the average of all TF-IDF values) as the core words of the collection. These phrases can better reflect the content of each document in this category text set.

[0085] Retrieve the creation time of all texts, and determine the coverage of the category text set based on the earliest and latest times where the time is relatively dense. Record this as the coverage time limit, and integrate the time range to form a time pre-sequence for the category text set. The time pre-sequence can better assist retrieval during search queries.

[0086] Retrieve the file names, index terms and file addresses of all texts, sort the index terms of each text according to the strokes of the file name, and form a retrieval appendix for the category text set. This can help us lock the corresponding storage location more quickly when the text is retrieved.

[0087] The description text, time sequence and search appendix are sorted to obtain the index document of the category text set. All the core words of the collection and the collection time limit of each text in it are sorted to finally obtain the index entry of the category text set.

[0088] First, construct the index documents and corresponding index entries of all category text sets, and then organize the index entries to obtain a high-level index table.

[0089] Figure 6 A schematic diagram of indexing documents is shown, such as Figure 6 As shown, the index document includes: time preamble, descriptive text and retrieval appendix.

[0090] Figure 7 A schematic diagram of a target index table is shown, Figure 7 As shown, the target index table includes: the name of the category text set, the collection time (collection time limit), the core words of the collection and their IF-IDF values, and the index documents.

[0091] like Figure 8 As shown, in actual application scenarios, when users need to find a file in massive files stored in the cloud, most of them will enter some sentences used to describe the file in the search bar (referred to as query sentences), and these sentences need to be analyzed in the cloud to extract the stem information of these sentences.

[0092] These sentences are preprocessed, including word segmentation, stop word removal, and stem (keyword) extraction. The specific operation process is the same as the steps in extracting valid phrases.

[0093] Secondly, based on the similarity between the stem and the core words of the corpus, the category text set that may contain the files required by the user is screened out in the high-level index table:

[0094] According to the extracted stem content, the core words of the text collections of each category are compared.

[0095] If the similarity correlation between all the core words of a certain category text set and many stems is relatively low, then the many files corresponding to the category text set are likely not the files that the user wants.

[0096] The above similarity correlation can be described based on the cosine similarity between the core word of the corpus and the stem, combined with the TF-IDF value of the core word of the corpus: Calculate the similarity correlation evaluation P between the core word w of the corpus and the stem v w,v

[0097]

[0098] Where TI w represents the TF-IDF value of the core word w in the corpus; and Represent the word vectors of the core word w and stem v of the corpus respectively; Represents the cosine similarity between two word vectors.

[0099] Similarity correlation evaluation is greater than or equal to the threshold Y p , default Y p =0.6, it is believed that the core word w and the stem v have a high similarity correlation. Otherwise, the similarity correlation between the two is low.

[0100] ② If the similarity correlation of some stems of some core words in a certain category text set is high, then the corresponding files contained in the category text set may contain the files that the user wants, and it is recorded as the text set to be analyzed.

[0101] In addition, the time description in the query statement can be combined with the collection time of the category text set to assist in screening the category text set that meets the conditions.

[0102] Traverse all the category text sets, analyze the similarity and correlation between the core words of the text sets and many stems, and obtain many text sets to be analyzed.

[0103] Then, search for documents related to the query statement and stem in the search appendix of the corresponding index document:

[0104] The more semantically similar the word stem in the query statement is to the keywords and custom tags in the index terms, and the more similar keywords there are, the more the file corresponding to the index terms meets the requirements of the query statement; and the more semantically similar the query statement is to the file name, the more the file meets the requirements of the query statement.

[0105] Construct the compatibility evaluation F between the index terms of the file η and the query statement τ η :

[0106]

[0107] In the formula, Represents the word vector of phrase x in the index entry of document η; x η Represents the total number of keywords and custom tags in the index entries of file η; Represents the word vector of the stem y in the query sentence τ; y v Indicates the total number of stems in the query statement; Indicates the TF-IDF value of keyword x in the index entry of file η. Indicates the similarity between the file name of file η and the name of query statement τ.

[0108] Traverse all the files in the text set to be analyzed, calculate the compatibility evaluation between each file and the query statement, and then select the 15% (preset value) with the highest evaluation as the files that best meet the query statement requirements according to the size of the compatibility evaluation (from large to small) and mark them as relevant files.

[0109] When performing cloud storage: Before performing cloud storage, first extract the keywords of the file and combine them with the relevant information of the file to form the index terms of the text; then, based on the similarity between the index terms of the text and the index items, store the file in the corresponding category text set.

[0110] When performing a search: Based on the similarity between the query statement and the search terms, a considerable portion of the category text sets with low similarity can be removed, and then the remaining category text sets with high similarity are searched for the next step. Based on the similarity between the query statement and the index terms, several related files with high similarity are finally determined, and then the related files are returned.

[0111] It is understandable that the data query method improved by the embodiment of the present application forms index entries corresponding to each data text by analyzing the relevant information of each text that needs to be stored in the cloud and custom tags. By analyzing the similarity of the content of each data text and the relevance of the content of the index entry, a high-level index entry and index structure are constructed. The search statement is analyzed and processed, and its similarity and relevance with the optimized index are calculated, and files related to the search statement are searched in the search appendix, and finally the most relevant first few cloud storage files are returned.

[0112] Another data query method is also provided in the embodiment of the present application, including: S1, data acquisition preprocessing, obtaining the text information of each stored text that needs to be stored in the cloud; wherein the text information includes the file name, user-defined tags and text content; determining all keyword groups in the text information of each stored text based on the keyword extraction method; S2, category text set division, according to the similarity of the text information and keywords between the stored texts in the semantic space, all the stored texts are divided into each category text set, so that the semantically related or similar stored texts are divided into an index entry according to the category text set, which is convenient for subsequent accurate and efficient retrieval query. S3, similarity association evaluation value calculation, after constructing a high-level index table according to each category text set, the semantic similarity between the search core phrases representing the key information corresponding to the search statement and the collection core phrases representing the subject information corresponding to each category text set is combined to calculate the similarity association evaluation value, so as to perform more accurate search statement matching according to the similarity association evaluation value. S4, search statement query, obtain the text to be analyzed that is related to the semantics corresponding to the search statement based on the similarity association evaluation value, and further perform a more accurate association retrieval analysis on each text to be analyzed based on the semantics of the search statement, so that the query-related text obtained by the retrieval query is more accurate.

[0113] The data query method improved in the embodiment of the present application can efficiently search for the required text under the condition of high encryption level of cloud storage. Under the premise of ensuring the accuracy of text search, it can effectively reduce the time consumption of information in the process of network transmission compared with general methods, and uses an optimized retrieval method, which effectively reduces the search time consumed on the cloud server, and also supports multi-keyword search. It can effectively deal with the situation of changes in index tables caused by frequent changes in massive data; the optimized index structure can effectively improve the retrieval efficiency of retrieval in many files. At the same time, the targeted retrieval method also supports multi-keyword retrieval during retrieval, and does not involve the relevant content of the file during the retrieval process, which can effectively ensure the confidentiality of the relevant files.

[0114] Fig. 9 is a structural diagram of a data query device according to an embodiment of the present application, such as Fig. 9 As shown, the device comprises:

[0115] A receiving module 90 is used to receive a query statement, where the query statement is used to query target data from multiple data texts stored in the cloud;

[0116] An acquisition module 92 is used to acquire a target index table of multiple category text sets in the cloud storage, wherein the target index table at least contains a core word of each category text set, wherein the core word of the text set is used to characterize the subject information of the category text set, and the multiple category text sets are divided according to the text content of the multiple data texts and the keywords of the multiple data texts;

[0117] The query module 94 is used to obtain the similarity between the query statement and the core words of the text set of each category in the target index table, and query the target data from the multiple data texts according to the similarity.

[0118] Through the above-mentioned data query device, it is achieved by receiving a query statement, the query statement is used to query target data from multiple data texts stored in the cloud; obtaining a target index table of multiple category text sets in the cloud storage, the target index table at least contains the corpus core words of each category text set, the corpus core words are used to characterize the main information of the category text set, and the multiple category text sets are divided according to the text content of the multiple data texts and the keywords of the multiple data texts; obtaining the similarity between the query statement and the corpus core words of each category text set in the target index table, and querying the target data from the multiple data texts according to the similarity, thereby achieving the purpose of dividing semantically related or similar data texts into one index entry according to the category text set, thereby achieving the purpose of accurate and efficient information retrieval, thereby achieving the technical effect of improving information retrieval efficiency, and thereby solving the technical problem of low information retrieval efficiency in related technologies.

[0119] The acquisition module 92 in the above-mentioned data query device includes: an acquisition submodule, which is used to acquire the target index table of multiple category text sets in the cloud storage, including: constructing the index term of each data text according to the word frequency-inverse document frequency corresponding to the keywords of the multiple data texts, the file name of each data text, the creation time of each data text and the custom tag of each data text, wherein the custom tag is used to characterize the description information of the data text; performing word segmentation processing on each data text to obtain multiple phrases corresponding to each data text; mapping the multiple phrases corresponding to each data text to multiple word vectors, and determining the content similarity between each data text in the multiple data texts according to the multiple word vectors corresponding to each data text; determining the integration possibility index between each data text in the multiple data texts based on the content similarity between each data text in the multiple data texts and the similarity between the index terms of each data text; combining each data text in the multiple data texts in pairs based on the integration possibility index between each data text to obtain multiple groups of data texts; combining the multiple groups of data texts into multiple category text sets; generating the target index table based on the multiple category text sets.

[0120] The acquisition submodule also includes: an index term similarity unit, which is used to obtain the similarity between the file names of each data text; obtain the similarity between the custom tags of each data text; and determine the similarity between the index terms of each data text based on the similarity between the custom tags and the similarity between the file names of each data text.

[0121] The index term similarity unit includes: a file name subunit and a custom tag subunit, wherein the file name subunit is used to obtain the similarity between the file names of each data text, including: segmenting the file name of each data text to obtain multiple phrases corresponding to the file name, and removing stop words in the multiple phrases corresponding to the file name to obtain valid words corresponding to the file name; summing the word vectors of the valid words corresponding to the file name to obtain the sentence vector corresponding to the file name of each data text; and determining the similarity between the file names of each data text according to the sentence vector corresponding to the file name of each data text.

[0122] The custom label subunit is used to obtain the similarity between the custom labels of each data text, including: calling the custom label of each data text, and respectively calculating the cosine similarity between the custom labels of each data text; constructing a similarity matrix of the custom labels between two data texts in the multiple data texts according to the cosine similarity between the custom labels of each data text, and the similarity matrix is ​​used to characterize the cosine similarity between the multiple custom labels corresponding to the two data texts; extracting the maximum value of each column and the value of the column corresponding to the maximum value from the similarity matrix; determining the cosine similarity between the multiple custom labels corresponding to the two data texts based on the maximum value of each column and the value of the column corresponding to the maximum value.

[0123] The acquisition module 92 also includes: a combination submodule, which is used to combine the multiple groups of data texts into multiple category text sets, including: combining two data texts whose integration possibility index is higher than a preset threshold into a data text combination; obtaining the integration possibility index of each pair of data text combinations in the multiple groups of data text combinations; combining the multiple groups of data text combinations according to the integration possibility index of each pair of data text combinations to obtain the multiple category text sets.

[0124] The acquisition module 92 also includes: a corpus core word submodule, which is used to obtain the index terms of each data text in each category text set; determine the description text of each category text set based on the file name, keywords and custom tags in the index terms of each data text; obtain the word frequency-inverse document frequency of each phrase in the description text of each category text set, and determine the phrase in the description text whose word frequency-inverse document frequency is greater than the preset word frequency-inverse document frequency as the corpus core word.

[0125] The corpus core word submodule includes: a target index table unit, which is used to generate the target index table based on the multiple category text sets, including: obtaining the creation time of each data text, determining the collection period of each category text set according to the creation time, and determining the time prefix of each category text set according to the collection period; sorting the index entries of each data text according to the number of strokes of the file name of each data text in each category text set to obtain the retrieval appendix of each category text set; combining the description text, the time prefix and the retrieval appendix to obtain the index document of each category corpus; determining the target index table based on the index document of each category corpus.

[0126] It should be noted that Fig. 9 The data query device shown is used to perform Figure 2 The data query method shown, therefore the relevant explanations in the above data query method are also applicable to the data query device, and will not be repeated here.

[0127] An embodiment of the present application also provides a computer device, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor is connected to the memory and is used to execute the above-mentioned data query method.

[0128] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored computer program, wherein a device where the non-volatile storage medium is located executes the above-mentioned data query method by running the computer program.

[0129] An embodiment of the present application also provides a computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data query method in the present application are implemented.

[0130] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the data query method in the present application.

[0131] The embodiment of the present application also provides a computer program, which, when executed by a processor, implements the steps of the data query method in the present application.

[0132] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0133] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0134] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0135] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0136] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0137] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0138] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data query method, characterized in that: include: Receive a query statement, where the query statement is used to search for target data from a plurality of data texts stored in the cloud; Obtain a target index table of multiple category text sets in the cloud storage, wherein the target index table contains at least a core word of each category text set, wherein the core word of the text set is used to characterize the subject information of the category text set, and the multiple category text sets are divided according to the text content of the multiple data texts and the keywords of the multiple data texts; The similarity between the query statement and the core words of the text set of each category in the target index table is obtained, and the target data is queried from the multiple data texts according to the similarity.

2. The method according to claim 1, characterized in that Obtaining a target index table of multiple category text sets in the cloud storage includes: constructing an index entry for each data text according to the word frequency-inverse document frequency corresponding to the keywords of the multiple data texts, the file name of each data text, the creation time of each data text, and a custom tag of each data text, wherein the custom tag is used to represent the description information of the data text; Perform word segmentation on each data text to obtain multiple phrases corresponding to each data text; Mapping the multiple phrases corresponding to each data text into multiple word vectors, and determining the content similarity between each data text in the multiple data texts according to the multiple word vectors corresponding to each data text; Determining an integration possibility index between each of the plurality of data texts based on the content similarity between each of the plurality of data texts and the similarity between the index terms of each of the plurality of data texts; Combining each of the plurality of data texts in pairs based on the integration possibility index between each of the data texts to obtain a plurality of groups of data texts; Combining the multiple groups of data texts into multiple category text sets; The target index table is generated based on the multiple category text sets.

3. The method according to claim 2, characterized in that The method further comprises: Obtaining the similarity between the file names of each data text; Obtaining the similarity between the custom tags of each data text; The similarity between the index terms of each data text is determined based on the similarity between the custom tags and the similarity between the file names of each data text.

4. The method according to claim 3, characterized in that Obtaining the similarity between the file names of each data text, including: Segmenting the file name of each data text to obtain multiple phrases corresponding to the file name, and removing stop words from the multiple phrases corresponding to the file name to obtain valid words corresponding to the file name; Summing the word vectors of the valid words corresponding to the file name to obtain the sentence vector corresponding to the file name of each data text; The similarity between the file names of each data text is determined according to the sentence vector corresponding to the file name of each data text.

5. The method according to claim 3, characterized in that: Obtaining the similarity between the custom tags of each data text, including: Retrieve the custom label of each data text, and calculate the cosine similarity between the custom labels of each data text; Constructing a similarity matrix of the custom labels between two data texts in the plurality of data texts according to the cosine similarity between the custom labels of each data text, wherein the similarity matrix is ​​used to represent the cosine similarity between the plurality of custom labels corresponding to the two data texts; Extracting the maximum value of each column and the value of the column corresponding to the maximum value from the similarity matrix; The cosine similarity between the multiple custom tags corresponding to the two data texts is determined based on the maximum value of each column and the value of the column corresponding to the maximum value.

6. The method according to claim 2, characterized in that Combining the multiple groups of data texts into multiple category text sets, including: Combining two data texts whose integration possibility index is higher than a preset threshold into a data text combination; Obtaining integration possibility indicators of two-by-two data text combinations in the plurality of data text combinations; The plurality of data text combinations are combined according to the integration possibility index of the pairwise data text combinations to obtain the plurality of category text sets.

7. The method according to claim 1, characterized in that The method further comprises: Get the index entry of each data text in each category text set; Determine the description text of each category text set based on the file name, keywords and custom tags in the index entry of each data text; The word frequency-inverse document frequency of each phrase in the description text of each category text set is obtained, and the phrase in the description text whose word frequency-inverse document frequency is greater than a preset word frequency-inverse document frequency is determined as the core word of the corpus.

8. The method according to claim 7, characterized in that Generating the target index table based on the multiple category text sets includes: Obtaining the creation time of each data text, determining the collection period of each category text set according to the creation time, and determining the time pre-order of each category text set according to the collection period; Sorting the index terms of each data text in each category text set according to the number of strokes of the file name of each data text in each category text set to obtain a search appendix of each category text set; The description text, the time preamble and the search appendix are combined to obtain the index document of each category collection; The target index table is determined based on the index documents of each category corpus.

9. A data query device, characterized in that: include: A receiving module, used for receiving a query statement, wherein the query statement is used for searching target data from a plurality of data texts stored in the cloud; An acquisition module, used for acquiring a target index table of a plurality of category text sets in the cloud storage, wherein the target index table at least contains a core word of each category text set, wherein the core word of the text set is used to characterize the subject information of the category text set, and the plurality of category text sets are divided according to the text content of the plurality of data texts and the keywords of the plurality of data texts; The query module is used to obtain the similarity between the query statement and the core words of the text set of each category in the target index table, and query the target data from the multiple data texts according to the similarity.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; The processor is connected to the memory and is used to execute the data query method described in any one of claims 1 to 8.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the data query method described in any one of claims 1 to 8 is implemented.

Images