Data retrieval method, system, device, and storage medium

Abstract

The application discloses a data retrieval method, system, device and storage medium, and is applied to a retrieval system constructed by a search engine server and a database. The data retrieval method comprises the following steps: acquiring a to-be-retrieved field and a calculation function; returning a data table row key according to the to-be-retrieved field and the calculation function in combination with a pre-established index relationship; sending the data table row key to the database, and receiving retrieval result information returned by the database according to the data table row key. The index relationship is pre-established in the search engine server, the to-be-retrieved field and the calculation function are combined, and retrieval is performed on the index relationship, so that the demand of multi-condition retrieval can be met, and the intelligent degree of query is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a data retrieval method, system, device and storage medium. BACKGROUND

[0002] HBase is a distributed, column-oriented, scalable open source database. In the prior art, when performing big data retrieval based on HBase, especially for massive data retrieval, the retrieval process of HBase non-row key rowkey will perform full table scanning, and thus the retrieval performance is low and the query speed is slow. Therefore, in order to meet the high concurrency and high efficiency of big data retrieval, the database table row key is often used to sort and store data, and the row key is used to realize high-speed retrieval within milliseconds.

[0003] In the prior art, when performing big data retrieval based on rowkey, there are mainly two retrieval schemes, the first one is based on a rowkey for retrieval, which supports high concurrency and high retrieval efficiency, but the disadvantage is that the retrieval scene is simple and limited, and does not support complex combination scene retrieval; the second one is based on two-level index for retrieval, which expands the rowkey design, such as rowkey=X|Y, rowkey=Y|Z, rowkey=X|Z, which is suitable for retrieval of two-by-two combined query conditions, and still supports high concurrency, but this retrieval scheme still has the limitation that it cannot meet the retrieval demand of arbitrary combination conditions.

[0004] Therefore, in view of the problem that the above two methods cannot meet the data retrieval of arbitrary combination conditions, it is necessary to propose a data retrieval scheme to solve the problem. SUMMARY

[0005] The main purpose of the present application is to provide a data retrieval method, system, device and storage medium, which aims to solve the problem that the prior art cannot meet the data retrieval of arbitrary combination conditions.

[0006] To achieve the above purpose, the present application provides a data retrieval method applied to a search engine server, wherein the search engine server is connected with a database, and the data retrieval method comprises:

[0007] obtaining a to-be-retrieved field and a calculation function;

[0008] returning a data table row key according to the to-be-retrieved field and the calculation function in combination with a pre-established index relationship;

[0009] sending the data table row key to the database and receiving retrieval result information returned by the database according to the data table row key.

[0010] Optionally, the step of returning a data table row key according to the field to be searched and the calculation function in combination with a pre-established index relationship comprises:

[0011] According to the field to be searched and the calculation function, matching the index relationship;

[0012] In the case of not matching the index relationship, judging whether the calculation function is an atomic calculation;

[0013] If the calculation function is an atomic calculation, returning a prompt message that does not conform to the operation rule;

[0014] If the calculation function is not an atomic calculation, decomposing the calculation function into atomic calculations, and returning the data table row key according to the field to be searched and the decomposed atomic calculations in combination with the index relationship.

[0015] Optionally, the data searching method further comprises:

[0016] In the case of returning the data table row key, generating and returning a relationship between the field to be searched, the calculation function and the data table row key based on the data table row key.

[0017] Optionally, the step of matching the index relationship according to the field to be searched and the calculation function further comprises:

[0018] In the case of matching the index relationship, returning a corresponding data table row key according to the matched index relationship.

[0019] Optionally, the step of returning a data table row key according to the field to be searched and the calculation function in combination with a pre-established index relationship further comprises:

[0020] Establishing the index relationship, wherein the index relationship comprises an index relationship between a search field and a data table row key set, and an index relationship between a search field, a calculation function and a data table row key set; the data table row key set is composed of a plurality of data table row keys.

[0021] Optionally, the step of establishing the index relationship comprises:

[0022] Importing a plurality of data fields and a data table row key set collected in advance, and creating the index relationship between the search field and the data table row key set based on the association of the data fields and the data table row key set in the database;

[0023] Packaging a calculation function, and creating the index relationship between the search field, the calculation function and the data table row key set based on the association of the packaged calculation function, the data fields and the data table row key set in the database.

[0024] Optionally, before the steps of obtaining the fields to be searched and the calculation function, the method further comprises:

[0025] creating a search query page, wherein the search query page is used for a user to input the fields to be searched and the calculation function;

[0026] creating a representational state transfer (Rest) interface, wherein the Rest interface is used for connecting the search query page and the search engine server, and the Rest interface is further used for stipulating a method adopted by an http request and a link of the request.

[0027] Optionally, the search engine server comprises a cold data area and a hot data area, and after the steps of sending the data table row key to the database and receiving the search result information returned by the database according to the data table row key, the method further comprises:

[0028] generating a search record according to an index relationship used for searching;

[0029] updating the cold data area and / or the hot data area according to the search record, wherein a storage node of the cold data area is used for storing an index relationship lower than a preset search frequency, and a storage node of the hot data area is used for storing an index relationship higher than the preset search frequency.

[0030] Optionally, the step of updating the cold data area and / or the hot data area according to the search record comprises:

[0031] judging whether a search frequency of an index relationship in the search record is higher than the preset search frequency;

[0032] if the search frequency of the index relationship is higher than the preset search frequency, writing the index relationship into the hot data area; and / or,

[0033] if the search frequency of the index relationship is lower than the preset search frequency, migrating the index relationship to the cold data area.

[0034] Optionally, before the step of receiving the search result information returned by the database according to the data table row key, the method further comprises:

[0035] receiving, by the database, the data table row key sent by the search engine server;

[0036] searching the search result information according to the data table row key, and returning the search result information to the search engine server.

[0037] Optionally, before the step of receiving the data table row key sent by the search engine server, the method further comprises:

[0038] importing the original data file into the database;

[0039] extracting a data field based on the original data file;

[0040] pre-operating the data field based on a calculation function to obtain a field operation result;

[0041] establishing a data table row key set based on the field operation result;

[0042] sending the data field, the calculation function and the data table row key set to the search engine server, so that the search engine server creates an index relationship between the retrieval field, the calculation function and the data table row key set.

[0043] The embodiment of the application further provides a data retrieval system, which comprises a search engine server and a database; wherein the search engine server comprises an acquisition module, an index module and a result module;

[0044] The acquisition module is used for acquiring a to-be-retrieved field and a calculation function;

[0045] The index module is used for returning a data table row key according to the to-be-retrieved field and the calculation function in combination with a pre-established index relationship;

[0046] The result module is used for sending the data table row key to the database and receiving retrieval result information returned by the database according to the data table row key;

[0047] The database is used for receiving the data table row key sent by the search engine server, retrieving the retrieval result information according to the data table row key, and returning the retrieval result information to the search engine server.

[0048] The embodiment of the application further provides a data retrieval device, which comprises a memory, a processor and a data retrieval program stored in the memory and executable on the processor, and the data retrieval program implements the steps of the data retrieval method when executed by the processor.

[0049] The embodiment of the application further provides a computer readable storage medium, which stores a data retrieval program, and the data retrieval program implements the steps of the data retrieval method when executed by a processor.

[0050] The data retrieval method, system, device and storage medium provided by the embodiments of the present application obtain a to-be-retrieved field and a calculation function through a search engine server; return a data table row key according to the to-be-retrieved field and the calculation function in combination with a pre-established index relationship; send the data table row key to the database, and receive retrieval result information returned by the database according to the data table row key. Based on the present application, any combination of the to-be-retrieved field and the calculation function is supported, and sufficient flexibility is provided; by pre-establishing an index relationship in the search engine server, the retrieval is performed in combination with the retrieval condition composed of any combination and the index relationship, the demand for multi-condition retrieval can be met, and the intelligent degree of query is improved; meanwhile, the database retrieval is performed based on the data table row key returned by the index relationship, full table scanning is not needed, the query time is reduced, and the query efficiency is improved.

[0051] Compared with the prior art, the data retrieval system combining the search engine server and the database constructed based on the present application can realize the second-level response performance of massive data query, has the technical advantages of system framework flexibility, support for large amount of user data operation, fast reading speed and strong scalability. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 The figure is a functional module schematic diagram of the data retrieval device to which the present application belongs;

[0053] Figure 2 The figure is a flow schematic diagram of the first exemplary embodiment of the data retrieval method of the present application;

[0054] Figure 3 The figure is a flow schematic diagram of the second exemplary embodiment of the data retrieval method of the present application;

[0055] Figure 4 The figure is a flow schematic diagram of the third exemplary embodiment of the data retrieval method of the present application;

[0056] Figure 5 The figure is a flow schematic diagram of the fifth exemplary embodiment of the data retrieval method of the present application;

[0057] Figure 6 The figure is a flow schematic diagram of the sixth exemplary embodiment of the data retrieval method of the present application;

[0058] Figure 7 The figure is a flow schematic diagram of the seventh exemplary embodiment of the data retrieval method of the present application;

[0059] Figure 8 The figure is an execution effect schematic diagram of the exemplary embodiment of the data retrieval method of the present application.

[0060] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0061] It should be understood that the specific embodiments described herein are merely exemplary and not intended to limit the application.

[0062] The main solution of the embodiment of the application is: obtaining a to-be-retrieved field and a calculation function; returning a data table row key according to the to-be-retrieved field and the calculation function in combination with a pre-established index relationship; sending the data table row key to the database, and receiving retrieval result information returned by the database according to the data table row key. Based on the solution of the application, any combination of the to-be-retrieved field and the calculation function is supported, and sufficient flexibility is provided. By pre-establishing an index relationship in a search engine server, the index relationship is combined with a retrieval condition composed of any combination to perform retrieval, so that the demand for multi-condition retrieval is met, and the intelligent degree of query is improved. Meanwhile, the database is retrieved based on the data table row key returned by the index relationship, without performing full table scanning, so that the query time is reduced and the query efficiency is improved.

[0063] Compared with the prior art, the data retrieval system combining the search engine server and the database constructed based on the solution of the application can realize second-level response performance of massive data query, has the technical advantages of system framework flexibility, support for large amount of user data operation, fast reading speed, and strong scalability.

[0064] Specifically, referring to Figure 1 , Figure 1 is a schematic diagram of a functional module of an apparatus to which the data retrieval device of the application belongs. The data retrieval device can be a device independent of the apparatus and capable of data retrieval, which can be carried on the apparatus in the form of hardware or software.

[0065] In the embodiment, the apparatus to which the data retrieval device belongs at least includes an output module 110, a processor 120, a memory 130, and a communication module 140.

[0066] The memory 130 stores an operating system and a data retrieval program, and the data retrieval device can store the to-be-retrieved field and the calculation function obtained, the index relationship pre-established, the data table row key returned according to the to-be-retrieved field and the calculation function in combination with the index relationship pre-established, and the retrieval result information obtained according to the data table row key in the memory 130; the output module 110 can be a display screen and the like. The communication module 140 can include a satellite communication module, a WIFI module, a mobile communication module, and a Bluetooth module, and the like, and communicates with external devices or servers through the communication module 140.

[0067] When the data retrieval program in the memory 130 is executed by the processor, the following steps are implemented:

[0068] obtaining a field to be searched and a calculation function;

[0069] returning a data table row key according to the field to be searched and the calculation function in combination with a pre-established index relationship;

[0070] sending the data table row key to the database and receiving search result information returned by the database according to the data table row key.

[0071] Further, the data search program in the memory 130, when executed by the processor, further implements the following steps:

[0072] matching the index relationship according to the field to be searched and the calculation function;

[0073] if the index relationship is not matched, judging whether the calculation function is atomic calculation;

[0074] if the calculation function is atomic calculation, returning prompt information that does not conform to the operation rule;

[0075] if the calculation function is not atomic calculation, decomposing the calculation function into atomic calculation, and returning the data table row key according to the field to be searched and the decomposed atomic calculation in combination with the index relationship.

[0076] Further, the data search program in the memory 130, when executed by the processor, further implements the following steps:

[0077] in the case of returning the data table row key, generating and returning the relationship between the field to be searched, the calculation function and the data table row key based on the data table row key.

[0078] Further, the data search program in the memory 130, when executed by the processor, further implements the following steps:

[0079] in the case of matching the index relationship, returning the corresponding data table row key according to the matched index relationship.

[0080] Further, the data search program in the memory 130, when executed by the processor, further implements the following steps:

[0081] establishing the index relationship, wherein the index relationship includes an index relationship between a search field and a data table row key set, and an index relationship between a search field, a calculation function and a data table row key set; the data table row key set is composed of a plurality of data table row keys.

[0082] Further, the data search program in the memory 130, when executed by the processor, further implements the following steps:

[0083] Importing a plurality of data fields and a data table row key set collected in advance, and creating an index relationship between the search field and the data table row key set based on an association of the data fields and the data table row key set in the database;

[0084] Packaging a calculation function, and creating an index relationship between the search field, the calculation function and the data table row key set based on an association of the data fields and the data table row key set in the database.

[0085] Further, the data retrieval program in the memory 130 also implements the following steps when executed by the processor:

[0086] Creating a search query page, wherein the search query page is used for the user to input the search field and the calculation function;

[0087] Creating a Representational State Transfer (Rest) interface, wherein the Rest interface is used to connect the search query page and the search engine server, and the Rest interface is also used to agree on a method used by an http request and a link of the request.

[0088] Further, the data retrieval program in the memory 130 also implements the following steps when executed by the processor:

[0089] Generating a search record according to the index relationship used for searching;

[0090] Updating the cold data area and / or the hot data area according to the search record, wherein the storage nodes of the cold data area are used to store index relationships below a preset search frequency, and the storage nodes of the hot data area are used to store index relationships above the preset search frequency.

[0091] Further, the data retrieval program in the memory 130 also implements the following steps when executed by the processor:

[0092] Judging whether a search frequency of the index relationship in the search record is higher than the preset search frequency;

[0093] If the search frequency of the index relationship is higher than the preset search frequency, writing the index relationship into the hot data area; and / or,

[0094] If the search frequency of the index relationship is lower than the preset search frequency, migrating the index relationship to the cold data area.

[0095] Further, the data retrieval program in the memory 130 also implements the following steps when executed by the processor:

[0096] Receiving the data table row key sent by the search engine server through the database;

[0097] The data table row key is retrieved according to the data table, and the search result information is returned to the search engine server.

[0098] Further, the data retrieval program in the memory 130, when executed by the processor, also implements the following steps:

[0099] Importing the original data file into the database;

[0100] Extracting the data field based on the original data file;

[0101] Preoperating the data field based on the calculation function to obtain the field operation result;

[0102] Establishing the data table row key set based on the field operation result;

[0103] Sending the data field, the calculation function and the data table row key set to the search engine server, so that the search engine server creates an index relationship between the search field, the calculation function and the data table row key set.

[0104] The embodiment obtains the to-be-retrieved field and the calculation function, returns the data table row key according to the to-be-retrieved field and the calculation function in combination with the pre-established index relationship, sends the data table row key to the database, and receives the search result information returned by the database according to the data table row key. Based on the scheme, any combination of the to-be-retrieved field and the calculation function is supported, and sufficient flexibility is provided. By pre-establishing the index relationship in the search engine server, the search is performed in combination with the index relationship and the search condition composed of any combination, the demand for multi-condition search is met, and the intelligent degree of the query is improved. At the same time, the database is searched based on the data table row key returned by the index relationship, without full table scanning, the query time is reduced, and the query efficiency is improved.

[0105] The data retrieval system architecture involved in the embodiment of the application includes at least a search engine server and a database, wherein the search engine server is connected with the database.

[0106] The search engine server is mainly used for establishing and storing an index relationship between a retrieval field and a rowkey set of a data table, and an index relationship between a retrieval field, a calculation function and the rowkey set; and is also used for matching a pre-established index relationship according to information required to be retrieved by a user, such as a retrieval field and a calculation function, and returning a corresponding data table rowkey. An element in the rowkey set of the data table is a data table rowkey. In the embodiment of the present application, the search engine server selects a SolrCloud cluster composed of a plurality of Solr search application servers.

[0107] The database is mainly used for storing underlying data, such as a database table in a standard format. A data row of the database table includes a data table rowkey. A corresponding data row content can be obtained by querying the data table rowkey. In the embodiment of the present application, the database is also used for pre-operating on a data field according to a set calculation rule, and forming a field operation result. The database in the embodiment of the present application selects an open source database HBase.

[0108] Based on the above system architecture but not limited to the above architecture, the method embodiment of the present application is proposed.

[0109] Referring to Figure 2 , Figure 2 is a flowchart of a first exemplary embodiment of a data retrieval method of the present application. The data retrieval method of the embodiment is applied to a search engine server connected with a database, and the data retrieval method includes:

[0110] In step S10, a retrieval field to be retrieved and a calculation function are obtained.

[0111] Specifically, when a user needs to retrieve data, a retrieval field to be retrieved and a calculation function are obtained. The retrieval field to be retrieved and the calculation function can be obtained in the following ways: reading a pre-stored retrieval field to be retrieved and a calculation function, obtaining a user input retrieval field to be retrieved and a calculation function through a provided search query page, receiving a retrieval field to be retrieved and a calculation function sent by a third party device or server, and the like. In the embodiment, the calculation function can include but is not limited to calculation functions such as avg, sum and count.

[0112] By combining a plurality of retrieval fields to be retrieved and calculation functions in any way, rich and complex retrieval conditions can be obtained.

[0113] In step S20, a data table rowkey is returned according to the retrieval field to be retrieved and the calculation function, and in combination with a pre-established index relationship.

[0114] Specifically, the index relationship is pre-established and stored in the search engine server, wherein the index relationship can include an index relationship between the retrieval field and the rowkey set, and can also include an index relationship between the retrieval field, the calculation function and the rowkey set. The element in the rowkey set is a data table row key rowkey.

[0115] The data table row key rowkey is included in the data row of the database table. By associating the index relationship supporting the multi-condition retrieval request with the data row, the association of the index relationship supporting the multi-condition retrieval request and the data table row key in the data row can be completed.

[0116] In the embodiment, the data table row key is returned according to the obtained retrieval field and calculation function, in combination with the established index relationship. More specifically, the obtained retrieval field and calculation function are matched with the established index relationship, such as the index relationship between the retrieval field and the rowkey set or the index relationship between the retrieval field, the calculation function and the rowkey set, and the data table row key rowkey in the matched index relationship is returned.

[0117] In step S30, the data table row key is sent to the database, and the retrieval result information returned by the database is received.

[0118] Then, the data table row key rowkey is provided to the database, and the retrieval result information is returned to the search engine server.

[0119] Specifically, the data table row key rowkey returned in the above step is sent to the database connected with the search engine server, so that the database performs retrieval according to the data table row key rowkey to obtain the retrieval result information. After the database performs retrieval according to the data table row key rowkey to obtain the retrieval result information, the retrieval result information is returned to the search engine server. Then, the search engine server receives the retrieval result information returned by the database.

[0120] In the embodiment, the data table row key is returned according to the obtained retrieval field and calculation function, in combination with the established index relationship. More specifically, the obtained retrieval field and calculation function are matched with the established index relationship, such as the index relationship between the retrieval field and the rowkey set or the index relationship between the retrieval field, the calculation function and the rowkey set, and the data table row key rowkey in the matched index relationship is returned. Based on the scheme, any combination of the retrieval field and the calculation function is supported, which has sufficient flexibility. By pre-establishing the index relationship in the search engine server and combining the retrieval condition composed of any combination with the index relationship for retrieval, the demand for multi-condition retrieval can be met, the intelligent degree of query is improved, and database retrieval based on the data table row key returned by the index relationship is performed without full table scanning, which reduces the query time and improves the query efficiency.

[0121] Further, referring to Figure 3 , Figure 3 is a flowchart of a second exemplary embodiment of the data retrieval method of the present application. Based on the above-mentioned Figure 2 embodiment shown in the above-mentioned embodiment, the step S20 of returning the data table row key according to the to-be-retrieved field and the calculation function in combination with the pre-established index relationship can include:

[0122] Step S201, matching the to-be-retrieved field and the calculation function with the index relationship.

[0123] Specifically, for the obtained to-be-retrieved field and calculation function, the to-be-retrieved field and the calculation function are matched with the established index relationship, such as matching the obtained to-be-retrieved field and calculation function with the established index relationship between the retrieval field and the rowkey set; or matching the obtained to-be-retrieved field and calculation function with the established index relationship between the retrieval field, the calculation function and the rowkey set.

[0124] Step S202, in the case of not matching the index relationship, judging whether the calculation function is an atomic calculation.

[0125] Specifically, in the case of not matching the index relationship between the to-be-retrieved field and the calculation function, it is judged whether the calculation function is an atomic calculation. The atomic calculation can include but is not limited to count, max, min, sum, avg, etc.

[0126] Step S203, if the calculation function is an atomic calculation, returning a prompt information that does not conform to the operation rule.

[0127] Step S204, if the calculation function is not an atomic calculation, decomposing the calculation function into atomic calculations, and returning the data table row key according to the to-be-retrieved field and the decomposed atomic calculations in combination with the index relationship.

[0128] Specifically, if the calculation function used for matching is an atomic calculation, a prompt information that does not conform to the operation rule is generated and returned.

[0129] If the calculation function used for matching is not an atomic calculation, the calculation function is decomposed into atomic calculations. According to the obtained to-be-retrieved field and the decomposed atomic calculations, the data table row key rowkey is returned in combination with the pre-established index relationship. More specifically, the obtained to-be-retrieved field and the decomposed atomic calculations are matched with the established index relationship, such as the index relationship between the established to-be-retrieved field and the rowkey set or the index relationship between the established to-be-retrieved field and the calculation function and the rowkey set, and the data table row key rowkey in the matched index relationship is returned.

[0130] Further, in the embodiment, the data retrieval method can further include:

[0131] In step S205, in the case of returning the data table row key, the relationship between the to-be-retrieved field, the calculation function and the data table row key is generated and returned based on the data table row key.

[0132] Specifically, after the data table row key rowkey is returned according to the to-be-retrieved field and the atomic calculation in combination with the established index relationship, the relationship between the to-be-retrieved field, the calculation function and the rowkey is generated and returned according to the returned rowkey. In order to calculate the original to-be-retrieved field and the calculation function and other retrieval data according to the generated relationship between the to-be-retrieved field, the calculation function and the rowkey together with the retrieved result information after the query result information is returned from the database according to the rowkey.

[0133] Further, in the embodiment, after the to-be-retrieved field and the calculation function are matched with the index relationship in step S201, the method can further include:

[0134] In step S206, in the case of matching the index relationship, the corresponding data table row key is returned according to the matched index relationship.

[0135] Specifically, when the to-be-retrieved field and the calculation function are matched with the established index relationship, such as the to-be-retrieved field and the calculation function are matched with the index relationship between the to-be-retrieved field and the rowkey set or the index relationship between the to-be-retrieved field and the calculation function and the rowkey set, if the index relationship is matched, the corresponding data table row key rowkey is obtained and returned according to the matched index relationship.

[0136] Further, referring to Figure 4 , Figure 4 is a flowchart of a third exemplary embodiment of the data retrieval method of the present application. Based on the above Figure 2 or Figure 3In the embodiment shown, in the embodiment, before returning the data table row key according to the index relationship established in advance in the step S20 according to the search field and the calculation function, the step S20 can further include:

[0137] In the step S11, the index relationship is established, wherein the index relationship includes the index relationship between the search field and the data table row key set, and the index relationship between the search field, the calculation function and the data table row key set; the data table row key set is composed of a plurality of data table row keys.

[0138] In the embodiment, the step S11 is implemented between the step S10 and the step S20, and in other embodiments, the step S11 can also be implemented before the step S10.

[0139] Compared with the above Figure 2 or Figure 3 In the embodiment shown, the embodiment further includes a scheme for establishing the index relationship.

[0140] Specifically, the step of establishing the index relationship can include:

[0141] Importing a plurality of data fields and a data table row key set collected in advance, and creating the index relationship between the search field and the data table row key set based on the association of the data fields and the data table row key set in the database;

[0142] Packaging the calculation function, and creating the index relationship between the search field, the calculation function and the data table row key set based on the association of the packaged calculation function, the data fields and the data table row key set in the database.

[0143] More specifically, a plurality of data fields and a data table row key set of a database are collected in advance. The collected data fields and the data table row key set are imported into the search engine server. The imported data fields are used to match the input search field to obtain the rowkey corresponding to the database table row to be searched. Then, the index relationship between the search field and the rowkey set is created according to the association of the imported data fields and the data table row key set rowkey in the database, i.e. according to the corresponding relationship of the data fields and the data table rowkey in the database.

[0144] The calculation function is packaged, such as avg, sum, count and other calculation functions suitable for specific queries. The index relationship between the search field, the calculation function and the rowkey set is created according to the association of the packaged calculation function, the data fields and the data table rowkey set rowkey in the database, i.e. according to the corresponding relationship of the calculation function, the data fields and the data table rowkey in the database.

[0145] After that, the index relationship between the created search field and the rowkey set, and the index relationship between the search field, the calculation function and the rowkey set are stored in the search engine server / mapped to the corresponding core processing unit, and the data retrieval interface and the query statement required for the foreground query are developed.

[0146] In an embodiment, the association relationship between the imported data field and the data table rowkey set rowkey in the database, and the association relationship between the calculation function, the data field and the data table rowkey set rowkey in the database can be known through the pre-operation process of the data field. That is, the data field is pre-operated according to the set calculation rule in the database, the field operation result is obtained and the corresponding data table rowkey rowkey is generated. According to the data field, the calculation rule and the data table rowkey corresponding to the field operation result involved in the process, the association relationship among the three is obtained.

[0147] Through the pre-operation of the data field in the database in advance, more complex index conditions are provided for the query side, so that it is not necessary to operate the input search field and calculation function again to return the search result each time, and the efficient return of the search result is achieved.

[0148] Further, based on the above embodiment, in this embodiment, before the search field and the calculation function are obtained in step S10, the following can also be included:

[0149] Creating a search query page, wherein the search query page is used for a user to input the search field and the calculation function;

[0150] Creating a representational state transfer (Rest) interface, wherein the Rest interface is used to connect the search query page and the search engine server, and the Rest interface is also used to agree on the method adopted by the http request and the link of the request.

[0151] Specifically, a search query page is created, which is used to provide a user to input the search field and the calculation function. A representational state transfer (Rest) interface is created. The search query page and the search engine server are connected through the created Rest interface, and the method adopted by the http request and the link of the request are agreed through the created Rest interface.

[0152] Further, in this embodiment, the search engine server can include a cold data area and a hot data area. The storage node Node of the cold data area is provided with low hardware configuration, which is used to store data with low search frequency; the storage node Node of the hot data area is provided with high hardware configuration, which is used to store data with high search frequency.

[0153] Referring to Figure 5 , Figure 5 FIG. 5 is a flowchart illustrating a fifth exemplary embodiment of a data retrieval method according to the present application. After the step S30 of sending the data table row key to the database and receiving the retrieval result information returned by the database according to the data table row key, the method can further include the following steps:

[0154] In step S40, a retrieval record is generated according to the index relationship used in the retrieval.

[0155] Specifically, after receiving the retrieval result information returned by the database, a retrieval record is generated according to the index relationship used in the retrieval, to record the retrieval frequency of the index relationship.

[0156] In step S50, the cold data area and / or the hot data area are updated according to the retrieval record, wherein the storage nodes of the cold data area are used to store the index relationship with a retrieval frequency lower than a preset retrieval frequency, and the storage nodes of the hot data area are used to store the index relationship with a retrieval frequency higher than the preset retrieval frequency.

[0157] Specifically, the cold data area and / or the hot data area in the search engine server are updated according to the generated retrieval record. More specifically, a retrieval frequency threshold, i.e., a preset retrieval frequency, is set in advance. The preset retrieval frequency is used to compare with the retrieval frequency of the index relationship in the retrieval record. According to the comparison between the retrieval frequency of the index relationship in the retrieval record and the preset retrieval frequency, the data stored in the cold data area and / or the hot data area in the search engine server are updated. The storage nodes of the cold data area are used to store the index relationship with a retrieval frequency lower than the preset retrieval frequency, and the storage nodes of the hot data area are used to store the index relationship with a retrieval frequency higher than the preset retrieval frequency.

[0158] Further, the step of updating the cold data area and / or the hot data area according to the retrieval record can include:

[0159] determining whether the retrieval frequency of the index relationship in the retrieval record is higher than the preset retrieval frequency;

[0160] if the retrieval frequency of the index relationship is higher than the preset retrieval frequency, writing the index relationship into the hot data area; and / or,

[0161] if the retrieval frequency of the index relationship is lower than the preset retrieval frequency, migrating the index relationship to the cold data area.

[0162] Specifically, it is determined whether the retrieval frequency of the index relationship in the generated retrieval record is higher than the preset retrieval frequency; if the retrieval frequency of the index relationship is higher than the preset retrieval frequency, the index relationship is written into the hot data area in the search engine server; and / or,

[0163] If the search frequency of the index relationship is lower than the preset search frequency, the index relationship is migrated to the cold data area in the search engine server. It should be noted that the index relationship in the cold data area is not written, but the index relationship with a search frequency lower than the preset search frequency is periodically migrated.

[0164] Exemplarily, for the search records generated daily, the index conditions in the search records in the last day are written to the hot data area, and a high hardware configuration is provided for the hot data area Node. For the index conditions not searched in the last day, they are periodically and automatically migrated to the cold data area, and a low hardware configuration is provided for the cold data area Node. Among them, the index conditions in the cold data area are not written, but the index conditions not searched in the last day are periodically and automatically migrated.

[0165] The embodiment scheme divides the cold data area and the hot data area in the search engine server, defines the index conditions with high-frequency search as hot data and writes them to the hot data area, and defines the index conditions with low-frequency search as cold data and migrates them to the hot data area, thereby realizing high-throughput cold and hot data exchange, and periodically updating the cold and hot data through timing scheduling. A high-specification machine is provided for the hot data Node, and a low-specification machine is provided for the cold data Node, so that the data search efficiency can be further improved under the condition of limited resources.

[0166] Reference Figure 6 , Figure 6 The flowchart of the sixth exemplary embodiment of the data search method of the present application is shown. Based on any of the above embodiments, the data search method can further include, before the step of receiving the search result information returned by the database according to the data table row key:

[0167] In step S301, the data table row key sent by the search engine server is received through the database.

[0168] Specifically, when the user searches for data, the data table row key sent by the search engine server is received through the database connected with the search engine server.

[0169] In step S302, the search result information is searched according to the data table row key, and the search result information is returned to the search engine server.

[0170] Specifically, the search result information is obtained by searching according to the received data table row key, and the search result information is returned to the connected search engine server.

[0171] The embodiment receives the data table row key sent by the search engine server through the above scheme, retrieves the search result information according to the data table row key, and returns the search result information to the search engine server. The embodiment of the application performs database retrieval based on the data table row key returned by the index relationship, does not need to perform full table scanning, reduces the query time, and improves the query efficiency.

[0172] Further, with reference to Figure 7 , Figure 7 The flowchart of the seventh exemplary embodiment of the data retrieval method of the application is shown. Based on the above Figure 6 embodiment, before the step S70 of receiving the data table row key sent by the search engine server through the database, the embodiment can further include:

[0173] Step S01: importing the original data file into the database.

[0174] Step S02: extracting the data field based on the original data file.

[0175] Specifically, the original data file is imported into the database, and the data field is extracted from the imported original data file.

[0176] Optionally, before importing the original data file into the database, the format of the original data file is standardized to form a database table. Then, the database table with the standardized format is imported into the database.

[0177] Step S03: pre-operating the data field based on the calculation function to obtain the field operation result.

[0178] Specifically, the data field in the database is pre-operated in combination with the calculation function, such as the calculation functions of avg, sum, count, max, min, etc. The data field meeting the corresponding calculation rule is pre-operated in combination with the calculation function to obtain the field operation result.

[0179] Step S04: establishing the data table row key set based on the field operation result.

[0180] Specifically, the corresponding data table row key rowkey set is established according to the field operation result obtained after the pre-operation of the calculation function. For example, for the data fields 001, 002, and 003, the pre-operation is performed in combination with the calculation function max to obtain the field operation result 003, and the corresponding data table row key rowkey set is established according to the field operation result 003.

[0181] Step S05, sending the data field, the calculation function and the data table row key set to the search engine server, so that the search engine server creates an index relationship between the search field, the calculation function and the data table row key set.

[0182] Specifically, the data field used in the pre-operation, the calculation function and the established data table row key rowkey set are sent to the search engine server, so that the search engine server creates an index relationship between the search field, the calculation function and the rowkey set according to the received data field, the calculation function and the data table row key rowkey set.

[0183] Exemplarily, after the calculation function pre-operation on the data field in the HBase database, a certain format of XML file is submitted to the Solr server through the http request, and the index relationship between the search field, the calculation function and the rowkey set which can support multi-condition query is established in Solr.

[0184] Further, in the embodiment, the step of retrieving the search result information according to the data table row key can include: querying the search result information according to the data table row key and the rowkey column index information of the database.

[0185] Specifically, the corresponding data row record is extracted according to the received data table row key; the corresponding data table column record is located and queried according to the data row record and the rowkey column index information of the database, as the search result information. Then, the queried search result information is returned to the corresponding search engine server.

[0186] Exemplarily, the effect of data retrieval based on the HBase database + SolrCloud cluster built by the method embodiment is as follows:

[0187] The number of basic data for retrieval and query is 2 billion, and the deployment environment is shown in Table 1:

[0188]

[0189] Table 1: Deployment environment

[0190] Among them, the execution effect is as shown in Figure 8 In the 2 billion basic data, three, five and eight Chinese character filtering conditions are set respectively, and the response time of each filtering condition is 4.4 seconds, 0.3 seconds and 0.2 seconds respectively when the display field is 10, 30, 50 and full field.

[0191] The embodiment provides more complex index conditions for the query side by preoperating on the data field and the calculation function in the database in advance, so that the inputted to-be-retrieved field and the calculation function do not need to be operated again to return the retrieval result each time retrieval is performed, and the efficient return of the retrieval result is achieved.

[0192] In addition, compared with the prior art, the data retrieval system constructed based on the scheme of the application and combining the search engine server and the database can achieve the second-level response performance of massive data query, has the technical advantages of system framework flexibility, support of large amount of user data operation, fast reading speed and strong scalability.

[0193] In addition, the embodiment of the application further provides a data retrieval system, which comprises a search engine server and a database.

[0194] The acquisition module is used for acquiring a to-be-retrieved field and a calculation function.

[0195] The index module is used for returning a data table row key according to the to-be-retrieved field and the calculation function in combination with a pre-established index relationship.

[0196] The result module is used for sending the data table row key to the database and receiving retrieval result information returned by the database according to the data table row key.

[0197] The database is used for receiving the data table row key sent by the search engine server, retrieving the retrieval result information according to the data table row key, and returning the retrieval result information to the search engine server.

[0198] The principle and implementation process of the embodiment of data retrieval are described above, and will not be repeated here.

[0199] In addition, the embodiment of the application further provides a data retrieval device, which comprises a memory, a processor and a data retrieval program stored in the memory and capable of running on the processor.

[0200] Since the data retrieval program is executed by the processor, all the technical schemes of the foregoing embodiments are adopted, and at least all the beneficial effects brought by all the technical schemes of the foregoing embodiments are achieved, and will not be repeated here.

[0201] In addition, the embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a data retrieval program, and the data retrieval program is executed by a processor to implement the steps of the data retrieval method.

[0202] Since the data retrieval program is executed by the processor, all the technical solutions of all the foregoing embodiments are adopted, and all the beneficial effects brought by all the technical solutions of all the foregoing embodiments are achieved, which will not be repeated here.

[0203] Compared with the prior art, the data retrieval method, system, device and storage medium provided by the embodiment of the present application have the following advantages: the data retrieval method, system, device and storage medium provided by the embodiment of the present application acquire a to-be-retrieved field and a calculation function, return a data table row key according to the to-be-retrieved field and the calculation function in combination with a pre-established index relationship, send the data table row key to the database, and receive retrieval result information returned by the database according to the data table row key. Based on the present application, any combination of the to-be-retrieved field and the calculation function is supported, and sufficient flexibility is provided. By pre-establishing an index relationship in a search engine server, the retrieval condition composed of any combination is combined with the index relationship to perform retrieval, so that the demand for multi-condition retrieval is met, and the intelligent degree of query is improved. Meanwhile, the database is retrieved based on the data table row key returned by the index relationship, without full table scanning, so that the query time is reduced and the query efficiency is improved.

[0204] Compared with the prior art, the data retrieval system combining the search engine server and the database constructed based on the present application can realize the second-level response performance of massive data query, has the technical advantages of system framework flexibility, support for large amount of user data operation, fast reading speed and strong scalability.

[0205] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or systems including a series of elements not only include those elements, but also include other elements not explicitly listed, or further include elements inherent to such processes, methods, articles or systems. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or system including the element.

[0206] The serial numbers of the above embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0207] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, can also be through hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art contribution can be embodied in the form of software products, the computer software product is stored in the above-mentioned storage medium (such as ROM / RAM, magnetic disc, optical disc), including a number of instructions to make a device execute the method of each embodiment of the present application.

[0208] The above is only the preferred embodiment of the present application, not the patent scope of the present application, any equivalent structure or equivalent process transformation using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data retrieval method, characterized by, The application is applied to a search engine server connected with a database, and the data retrieval method comprises the following steps: Obtaining a to-be-retrieved field and a calculation function; Returning a data table row key according to the to-be-retrieved field and the calculation function in combination with a pre-established index relationship; Sending the data table row key to the database and receiving retrieval result information returned by the database according to the data table row key; The step of returning the data table row key according to the to-be-retrieved field and the calculation function in combination with the pre-established index relationship comprises the following steps: Matching the to-be-retrieved field and the calculation function with the index relationship; In the case that no index relationship is matched, judging whether the calculation function is an atomic calculation; If the calculation function is an atomic calculation, returning prompt information that does not conform to an operation rule; If the calculation function is not an atomic calculation, decomposing the calculation function into atomic calculations, and returning the data table row key according to the to-be-retrieved field and the decomposed atomic calculations in combination with the index relationship; Further comprising the following steps: Importing an original data file into the database; Extracting a data field based on the original data file; Preoperating on the data field based on a calculation function to obtain a field operation result; Establishing a data table row key set based on the field operation result; Sending the data field, the calculation function and the data table row key set to the search engine server to create an index relationship between the retrieval field, the calculation function and the data table row key set.

2. The data retrieval method of claim 1, wherein, The data retrieval method further comprises the following steps: In the case that the data table row key is returned, generating and returning a relationship between the to-be-retrieved field, the calculation function and the data table row key based on the data table row key.

3. The data retrieval method of claim 1, wherein, After the step of matching the to-be-retrieved field and the calculation function with the index relationship, the method further comprises the following steps: In the case that an index relationship is matched, returning a corresponding data table row key according to the matched index relationship.

4. The data retrieval method of claim 1, wherein, Before the step of returning the data table row key according to the to-be-retrieved field and the calculation function in combination with the pre-established index relationship, the method further comprises the following steps: Establishing the index relationship, wherein the index relationship comprises an index relationship between a retrieval field and a data table row key set, and an index relationship between a retrieval field, a calculation function and a data table row key set; the data table row key set is composed of a plurality of data table row keys.

5. The data retrieval method of claim 4, wherein, The step of establishing the index relationship comprises the following steps: Importing a plurality of pre-collected data fields and a data table row key set, and creating the index relationship between the retrieval field and the data table row key set based on the association of the data fields and the data table row key set in the database; Packaging a calculation function, and creating the index relationship between the retrieval field, the calculation function and the data table row key set based on the association of the packaged calculation function, the data fields and the data table row key set in the database.

6. The data retrieval method of claim 1, wherein, Before the step of obtaining the to-be-retrieved field and the calculation function, the method further comprises the following steps: Creating a search query page, wherein the search query page is used for a user to input the to-be-retrieved field and the calculation function; A representational state transfer Rest interface is created, wherein the Rest interface is used to connect the search query page and the search engine server, and the Rest interface is also used to agree on a method adopted by an http request and a link of the request.

7. The data retrieval method of claim 1, wherein, The search engine server comprises a cold data area and a hot data area, and after the step of sending the data table row key to the database and receiving the search result information returned by the database according to the data table row key, the method further comprises the steps of: generating a search record according to an index relationship used for searching; updating the cold data area and / or the hot data area according to the search record, wherein a storage node of the cold data area is used to store an index relationship below a preset search frequency, and a storage node of the hot data area is used to store an index relationship above the preset search frequency.

8. The data retrieval method of claim 7, wherein, The step of updating the cold data area and / or the hot data area according to the search record comprises the steps of: judging whether a search frequency of an index relationship in the search record is higher than the preset search frequency; if the search frequency of the index relationship is higher than the preset search frequency, writing the index relationship into the hot data area; and / or if the search frequency of the index relationship is lower than the preset search frequency, migrating the index relationship to the cold data area.

9. The data retrieval method of claim 1, wherein, Before the step of receiving the search result information returned by the database according to the data table row key, the method further comprises the steps of: receiving the data table row key sent by the search engine server through the database; searching to obtain the search result information according to the data table row key, and returning the search result information to the search engine server.

10. A data retrieval system characterized by, The data search system comprises a search engine server and a database; wherein the search engine server comprises an acquisition module, an index module and a result module; The acquisition module is used to acquire a field to be searched and a calculation function; The index module is used to return a data table row key according to the field to be searched and the calculation function in combination with a pre-established index relationship; The result module is used to send the data table row key to the database, and receive search result information returned by the database according to the data table row key; The database is used to receive the data table row key sent by the search engine server, search to obtain the search result information according to the data table row key, and return the search result information to the search engine server; The step of returning the data table row key according to the field to be searched and the calculation function in combination with a pre-established index relationship comprises the steps of: matching the index relationship according to the field to be searched and the calculation function; in a case where the index relationship cannot be matched, judging whether the calculation function is an atomic calculation; if the calculation function is an atomic calculation, returning prompt information that does not conform to an operation rule; if the calculation function is not an atomic calculation, decomposing the calculation function into atomic calculations, and returning the data table row key according to the field to be searched and the decomposed atomic calculations in combination with the index relationship; The method further comprises the steps of: importing an original data file into the database; extracting a data field based on the original data file; pre-operate the data field based on a calculation function to obtain a field operation result; establish a data table row key set based on the field operation result; send the data field, the calculation function and the data table row key set to the search engine server, so that the search engine server creates an index relationship between the obtained search field, the calculation function and the data table row key set.

11. A data retrieval apparatus, characterized by comprising: The data retrieval device includes a memory, a processor and a data retrieval program stored on the memory and executable on the processor, and the data retrieval program implements the steps of the data retrieval method according to any one of claims 1-9 when executed by the processor.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a data retrieval program, and the data retrieval program implements the steps of the data retrieval method according to any one of claims 1-9 when executed by the processor.

Images