Method, system, device and storage medium for data writing and data querying

Abstract

The application discloses a data writing, data query method, system, device and storage medium, the data writing method comprises the following steps: writing the data to be written in the form of key-value pair into a key-value storage system; presetting a secondary index column; obtaining a first primary key set; establishing a first index relationship between the value of the secondary index column and the first primary key set, and writing the first index relationship into the key-value storage system; presetting a full-text index column; obtaining corresponding words in the full-text index column, and obtaining a second primary key set; establishing a second index relationship between the words and the second primary key set, and writing the second index relationship into the key-value storage system. The application realizes the demand of simultaneously supporting relational query and full-text retrieval, reduces the implementation difficulty and complexity, and solves the problems of easy to cause great waste of storage resources and network resources and easy to cause inaccurate full-text retrieval results in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a method, system, device and storage medium for data writing and data query. Background Art

[0002] Currently, relational databases and full-text search are two independent data processing methods. In other words, there is no existing data storage system that can handle both relational query and full-text search. In order to meet the needs of relational query and full-text search at the same time, the existing method is to integrate relational databases and full-text search through the following steps:

[0003] Deploy a relational database service to accept transactional data writes and queries; a full-text search service to accept analytical and retrieval full-text search requests; and a periodic data transmission and synchronization service to periodically synchronize data from the relational database's existing static data and incremental dynamic data to the full-text search service. Specifically, first query a copy of static data from the relational database and record the timestamp t1 when the data was queried. This static data is written to the full-text search service all at once. Then, starting from timestamp t1, all subsequent relational database log information is collected. Incremental data is generated based on the relational database's dynamic log information and periodically written to the full-text search service.

[0004] Therefore, the existing methods that can simultaneously support relational databases and full-text retrieval have the following defects: (1) Three sets of services need to be deployed, which is difficult and complex to implement; (2) The same data must be periodically transferred from one service to another just to meet different data usage and query requirements, which can easily cause a huge waste of storage resources and network resources; (3) Since data needs to be periodically synchronized and transmitted, and the existing synchronization cycles are relatively long, the data in the full-text retrieval service will have a large data delay compared to the data in the relational database, which further leads to inaccurate full-text retrieval results. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that the data storage system in the existing technology that can simultaneously meet the needs of relational query and full-text retrieval needs to deploy three sets of services, which has the disadvantages of great implementation difficulty and complexity, easy to cause great waste of storage resources and network resources, and easy to lead to inaccurate full-text retrieval results. The purpose is to provide a method, system, device and storage medium for data writing and data query.

[0006] The present invention solves the above technical problems through the following technical solutions:

[0007] The present invention provides a data writing method, the data writing method comprising:

[0008] Write the data to be written into the key-value storage system in the form of key-value pairs;

[0009] Wherein, the key-value storage system includes multiple data tables, and the key-value pairs include primary keys corresponding to the data tables;

[0010] Presetting a secondary index column in the data table for relational query;

[0011] Obtain a first primary key set corresponding to the value of the secondary index column;

[0012] Establishing a first index relationship between the value of the secondary index column and the first primary key set, and writing the first index relationship into the key-value storage system;

[0013] Presetting a full-text index column in the data table for full-text search;

[0014] Obtaining corresponding entries in the full-text index column, and obtaining second primary key sets corresponding to different data tables containing the entries;

[0015] A second index relationship is established between the entry and the second primary key set, and the second index relationship is written into the key-value storage system.

[0016] Preferably, before the step of writing the data to be written into the key-value storage system in the form of a key-value pair, the step further includes:

[0017] Sending a request to the key-value storage system to apply for writing the data to be written;

[0018] generating a write sequence number according to the request;

[0019] The write sequence number and the data to be written are written into a write-ahead log system.

[0020] Preferably, after the step of writing the data to be written into the key-value storage system in the form of a key-value pair and before the step of obtaining the corresponding entry in the full-text index column, the method further includes:

[0021] Determine whether the data to be written is successfully written into the key-value storage system; if so, continue to determine whether the data table has a secondary index column for relationship query; if so, execute the steps of establishing a first index relationship between the value of the secondary index column and the first primary key set, and write the first index relationship into the key-value storage system; continue to determine whether the data table has a full-text index column for full-text search; if so, execute the step of obtaining the corresponding entry in the full-text index column; or,

[0022] After the data to be written is successfully written into the key-value storage system, it is determined whether a full-text index column for full-text retrieval is provided in the data table; if so, the step of obtaining the corresponding entry in the full-text index column is executed, and it is further determined whether the data table is provided with a secondary index column for relational query; if so, the step of establishing a first index relationship between the value of the secondary index column and the first primary key set and writing the first index relationship into the key-value storage system is executed.

[0023] Preferably, the step of establishing a second index relationship between the entry and the second primary key set includes:

[0024] Lucene (a full-text search engine) is used to establish a second index relationship between the entry and the second primary key set.

[0025] The present invention also provides a data query method, which is implemented using the above-mentioned data writing method, and includes:

[0026] Get the filter conditions;

[0027] When the filtering condition includes the value in the secondary index column and the target term, the corresponding first primary key set is obtained according to the value in the secondary index column and the corresponding second primary key set is obtained according to the target term;

[0028] Taking the intersection of the first primary key set and the second primary key set as the target primary key set;

[0029] The corresponding target data is obtained by querying the target primary key set.

[0030] Preferably, when the filtering condition only includes the target primary key, the target key value corresponding to the target primary key is obtained;

[0031] Target data corresponding to the target primary key is obtained by querying from the key-value storage system according to the target key value.

[0032] Preferably, when the filtering condition only includes the values ​​in the secondary index column, the first primary key set corresponding to the values ​​in the secondary index column is obtained;

[0033] The corresponding target data is obtained by querying according to the first primary key set.

[0034] Preferably, when the filtering condition includes only the target term, the second primary key set corresponding to the target term is obtained;

[0035] The corresponding target data is obtained by querying the second primary key set.

[0036] The present invention also provides a data writing system, which includes a data writing module, a secondary index column preset module, a first primary key set acquisition module, a first index relationship establishment module, a first index relationship writing module, a full-text index column preset module, a second primary key set acquisition module, a second index relationship establishment module, and a second index relationship writing module;

[0037] The data writing module is used to write the data to be written into the key-value storage system in the form of key-value pairs;

[0038] Wherein, the key-value storage system includes multiple data tables, and the key-value pairs include primary keys corresponding to the data tables;

[0039] The secondary index column preset module is used to preset the secondary index column for relationship query in the data table;

[0040] The first primary key set acquisition module is used to acquire a first primary key set corresponding to the value of the secondary index column;

[0041] The first index relationship establishing module is used to establish a first index relationship between the value of the secondary index column and the first primary key set;

[0042] The first index relationship writing module is used to write the first index relationship into the key-value storage system;

[0043] The full-text index column preset module is used to preset the full-text index columns in the data table for full-text search;

[0044] The second primary key set acquisition module is used to acquire the corresponding entry in the full-text index column, and acquire the second primary key set corresponding to the different data tables containing the entry;

[0045] The second index relationship establishment module is used to establish a second index relationship between the entry and the second primary key set;

[0046] The second index relationship writing module is used to write the second index relationship into the key-value storage system.

[0047] Preferably, the data writing system further comprises a request sending module, a serial number obtaining module and a first writing module;

[0048] The request sending module is used to send a request to the key-value storage system to apply for writing the data to be written;

[0049] The serial number acquisition module is used to generate a write serial number according to the request;

[0050] The first writing module is used to write the write sequence number and the data to be written into a write-ahead log system.

[0051] Preferably, the data writing system further includes a judgment module;

[0052] The judgment module is used to judge whether the data to be written is successfully written into the key-value storage system. If it is successfully written, it continues to judge whether the data table is provided with a secondary index column for relationship query. If it is provided with the secondary index column, it calls the first index relationship establishment module and continues to judge whether the data table is provided with a full-text index column for full-text search. If it is provided with the full-text index column, it calls the second primary key set acquisition module; or,

[0053] The judgment module is used to determine whether the data to be written is successfully written into the key-value storage system. If it is successfully written, it continues to determine whether a full-text index column for full-text retrieval is provided in the data table. If the full-text index column is provided, the second primary key set acquisition module is called, and the judgment module continues to determine whether the data table is provided with a secondary index column for relationship query. If the secondary index column is provided, the first index relationship establishment module is called.

[0054] Preferably, the second index relationship establishment module is used to establish a second index relationship between the entry and the second primary key set using Lucene.

[0055] The present invention also provides a data query system, which is implemented by the above-mentioned data writing system. The data query system includes a filtering condition acquisition module, a target primary key acquisition module and a target data query module;

[0056] The filtering condition acquisition module is used to obtain filtering conditions;

[0057] The target primary key acquisition module is configured to, when the filtering condition includes a value in the secondary index column and a target term, acquire the corresponding first primary key set according to the value in the secondary index column and acquire the corresponding second primary key set according to the target term, and use the intersection of the first primary key set and the second primary key set as the target primary key set;

[0058] The target data query module is used to query and obtain corresponding target data according to the target primary key set.

[0059] Preferably, the target primary key acquisition module is used for acquiring a target key value corresponding to the target primary key when the filtering condition only includes the target primary key;

[0060] The target data query module is used to query the key-value storage system according to the target key value to obtain the target data corresponding to the target primary key.

[0061] Preferably, the target primary key acquisition module is configured to acquire the first primary key set corresponding to the values ​​in the secondary index column when the filtering condition only includes the values ​​in the secondary index column;

[0062] The target data query module is used to query and obtain corresponding target data according to the first primary key set.

[0063] Preferably, the target data query module is configured to obtain the second primary key set corresponding to the target term when the filtering condition only includes the target term;

[0064] The target data query module is used to query and obtain corresponding target data according to the second primary key set.

[0065] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data writing method when executing the computer program.

[0066] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data query method when executing the computer program.

[0067] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above-mentioned data writing method when executed by a processor.

[0068] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above-mentioned data query method when executed by a processor.

[0069] The positive progress effect of the present invention is:

[0070] In the present invention, when writing the data to be written, the data to be written is first written into the key-value storage system in one transaction, and then the secondary index writing and the inverted index writing are performed; when the data is queried, the primary key set that meets the conditions is queried based on the secondary index and the inverted index in combination with the filtering conditions, and then the final target data is queried, thereby achieving the needs of supporting relational query and full-text retrieval at the same time, reducing the difficulty and complexity of implementation, and overcoming the existing defects of easily causing great waste of storage resources and network resources and easily leading to inaccurate full-text retrieval results. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 This is a flow chart of the data writing method according to embodiment 1 of the present invention.

[0072] Figure 2 This is a flow chart of a data writing method according to embodiment 2 of the present invention.

[0073] Figure 3 This is a flow chart of the data query method according to embodiment 3 of the present invention.

[0074] Figure 4 This is a structural diagram of a data writing system according to embodiment 5 of the present invention.

[0075] Figure 5 This is a structural diagram of a data writing system according to embodiment 6 of the present invention.

[0076] Figure 6 This is a structural diagram of a data query system according to embodiment 7 of the present invention.

[0077] Figure 7 This is a structural diagram of an electronic device for implementing a data writing method according to embodiment 9 of the present invention. DETAILED DESCRIPTION

[0078] The present invention is further described below by way of examples, but the present invention is not limited to the scope of the examples.

[0079] Example 1

[0080] like Figure 1 As shown, the data writing method of this embodiment includes:

[0081] S101, writing the data to be written into the key-value storage system in the form of key-value pairs;

[0082] The key-value storage system includes multiple data tables, and the key-value pairs include primary keys corresponding to the data tables.

[0083] Specifically, the format corresponding to Key is: table ID: primary key ID: primary key value; the content of Value is a Byte array converted from a string consisting of all column names and column values ​​in a row of data.

[0084] RocksDB (a database software) is used to support the reading and writing of basic data in the key-value storage system. The strings corresponding to the above format actually stored in RocksDB are converted into Byte arrays for storage, so that the data occupies a relatively small memory capacity and ensures the speed of data transmission.

[0085] S102, presetting a secondary index column in a data table for relational query;

[0086] S103. Obtain a first primary key set corresponding to the value of the secondary index column;

[0087] S104: Establish a first index relationship between the value of the secondary index column and the first primary key set, and write the first index relationship into the key-value storage system;

[0088] In relational databases, secondary indexes are used to quickly find the corresponding primary key data structure based on the contents of non-primary key columns. Secondary index columns correspond to non-primary key columns in a table, and the values ​​of non-primary key columns are used to quickly locate the primary keys of all rows containing these values.

[0089] Specifically, the format of the secondary index is a Key-Value pair, where the Value is null (empty), and the key format is table ID: column ID: column value: primary key value.

[0090] Secondary indexes can also be implemented in memory through B-tree or B+ tree organization. The process of implementing through B-tree or B+ tree organization is an existing mature technology, so it will not be described in detail here.

[0091] In addition, the string corresponding to the above format stored in RocksDB is converted into a Byte array for storage.

[0092] S105, presetting a full-text index column in the data table for full-text search;

[0093] S106: Obtain the corresponding term in the full-text index column, and obtain the second primary key set corresponding to different data tables containing the term;

[0094] S107: Establish a second index relationship between the entry and the second primary key set, and write the second index relationship into the key-value storage system.

[0095] Specifically, based on the idea of ​​inverted index, a term is used to quickly find the primary key set of all rows whose specific columns contain the term or the primary key set corresponding to the document.

[0096] Considering that for a table, each insertion of a row of data may modify the primary key set corresponding to any term in the inverted index, the inverted index needs to be modified frequently. Therefore, it is not suitable to be placed in a key-value storage system, but needs to be managed and stored separately. In this embodiment, Lucene is used to establish a secondary index relationship between the entry and the second primary key set to achieve the management and storage of the inverted index.

[0097] In this embodiment, when writing data to be written, the data to be written is first written into the key-value storage system in one transaction, and then the secondary index and inverted index are written, and the transaction is submitted to complete the data writing operation, thereby achieving the need to support relational query and full-text retrieval at the same time, reducing the implementation difficulty and complexity, and overcoming the existing defects of easily causing great waste of storage resources and network resources and easily leading to inaccurate full-text retrieval results.

[0098] Example 2

[0099] like Figure 2 As shown, the data writing method of this embodiment is a further improvement of embodiment 1, specifically:

[0100] Before step S101, the following steps are also included:

[0101] S10101. Send a request to the key-value storage system to apply for writing the data to be written;

[0102] S10102. Generate a write sequence number according to the request;

[0103] S10103. Write the write sequence number and the data to be written into the write-ahead log system.

[0104] After step S101, the following steps are also included:

[0105] Determine whether the data to be written is successfully written into the key-value storage system. If it is successfully written, continue to determine whether the data table has a secondary index column for relationship query. If it has a secondary index column, execute step S104, and continue to determine whether the data table has a full-text index column for full-text search. If it has a full-text index column, execute step S106; or,

[0106] After the data to be written is successfully written into the key-value storage system, it is determined whether a full-text index column for full-text search is provided in the data table. If a full-text index column is provided, step S106 is executed, and it is further determined whether a secondary index column for relational query is provided in the data table. If a secondary index column is provided, step S104 is executed.

[0107] In other words, in this embodiment, the data to be written is first ensured to be successfully written into the key-value storage system, and then the secondary index information and full-text index information are written separately, thereby enabling the key-value storage system to support both relational queries and full-text searches. The order in which the secondary index information and full-text index information are written is not specifically specified.

[0108] In addition, if a write operation to the key-value storage system fails, a rollback operation is performed and the information of the write failure is recorded.

[0109] In this embodiment, when writing data to be written, the data to be written is first written into the key-value storage system in one transaction, and then the secondary index and inverted index are written, and the transaction is submitted to complete the data writing operation, thereby achieving the need to support relational query and full-text retrieval at the same time, reducing the implementation difficulty and complexity, and overcoming the existing defects of easily causing great waste of storage resources and network resources and easily leading to inaccurate full-text retrieval results.

[0110] Example 3

[0111] The data query method of this embodiment is implemented by using the data writing method of any one of Embodiments 1 or 2.

[0112] Specifically, if Figure 3 As shown, the data query method of this embodiment includes:

[0113] S201, obtaining filtering conditions;

[0114] S202: When the filtering condition includes a value in a secondary index column and a target term, a corresponding first primary key set is obtained according to the value in the secondary index column and a corresponding second primary key set is obtained according to the target term;

[0115] S203: Taking the intersection of the first primary key set and the second primary key set as the target primary key set;

[0116] S204: Query and obtain corresponding target data according to the target primary key set.

[0117] In this embodiment, the filtering conditions are combined and the primary key set that meets the conditions is queried based on the secondary index and the inverted index, and then the final target data is queried, thereby meeting the needs of supporting relational query and full-text retrieval at the same time, reducing the difficulty and complexity of implementation, and overcoming the existing defects of easily causing great waste of storage resources and network resources and easily leading to inaccurate full-text retrieval results.

[0118] Example 4

[0119] The data query method of this embodiment is a further improvement of embodiment 3. Specifically:

[0120] After step S201, the following steps are also included:

[0121] When the filtering condition only includes the target primary key, no secondary index is required, nor is a full-text search. In this case, the target key value corresponding to the target primary key is directly obtained.

[0122] According to the target key value, the target data corresponding to the target primary key is obtained from the key-value storage system. Alternatively, after step S201, the following steps are further included:

[0123] When the filtering condition only includes the values ​​in the secondary index column, no full-text search is required. In this case, the first primary key set corresponding to the values ​​in the secondary index column is obtained.

[0124] Querying the first primary key set to obtain the corresponding target data. Alternatively, after step S201, the method further includes:

[0125] When the filtering condition only includes the target term, no secondary index is required. In this case, the second primary key set corresponding to the target term is obtained.

[0126] The corresponding target data is obtained by querying the second primary key set.

[0127] In this embodiment, different filtering conditions are combined to obtain query results that match the filtering conditions, and queries on target data are independently implemented based on the target primary key, secondary index, and full-text index, thereby improving the existing data query method and enhancing the user data query experience.

[0128] Example 5

[0129] like Figure 4 As shown, the data writing system of this embodiment includes a data writing module 1, a secondary index column preset module 2, a first primary key set acquisition module 3, a first index relationship establishment module 4, a first index relationship writing module 5, a full-text index column preset module 6, a second primary key set acquisition module 7, a second index relationship establishment module 8 and a second index relationship writing module 9.

[0130] The data writing module 1 is used to write the data to be written into the key-value storage system in the form of key-value pairs;

[0131] The key-value storage system includes multiple data tables, and the key-value pairs include primary keys corresponding to the data tables.

[0132] Specifically, the format of Key is: table ID: primary key ID: primary key value; the content of Value is a Byte array converted from a string consisting of all column names and column values ​​in a row of data.

[0133] RocksDB is used to support the reading and writing of basic data in the key-value storage system. The strings corresponding to the above format actually stored in RocksDB are converted into Byte arrays for storage, so that the data occupies a relatively small memory capacity and ensures the speed of data transmission.

[0134] The secondary index column preset module 2 is used to preset the secondary index columns for relational query in the data table;

[0135] The first primary key set acquisition module 3 is used to acquire the first primary key set corresponding to the value of the secondary index column;

[0136] The first index relationship establishing module 4 is used to establish a first index relationship between the value of the secondary index column and the first primary key set;

[0137] The first index relationship writing module 5 is used to write the first index relationship into the key-value storage system;

[0138] In relational databases, secondary indexes are used to quickly find the corresponding primary key data structure based on the contents of non-primary key columns. Secondary index columns correspond to non-primary key columns in a table, and the values ​​of non-primary key columns are used to quickly locate the primary keys of all rows containing these values.

[0139] Specifically, the format of the secondary index is a Key-Value pair, where the Value is null (empty), and the key format is table ID: column ID: column value: primary key value.

[0140] Secondary indexes can also be implemented in memory through B-tree or B+ tree organization. The process of implementing through B-tree or B+ tree organization is an existing mature technology, so it will not be described in detail here.

[0141] In addition, the string corresponding to the above format stored in RocksDB is converted into a Byte array for storage.

[0142] The full-text index column preset module 6 is used to preset the full-text index columns for full-text search in the data table;

[0143] The second primary key set acquisition module 7 is used to acquire the corresponding entry in the full-text index column and acquire the second primary key sets corresponding to different data tables containing the entry;

[0144] The second index relationship establishing module 8 is used to establish a second index relationship between the entry and the second primary key set;

[0145] Specifically, the second index relationship establishing module is used to establish a second index relationship between the term and the second primary key set using Lucene.

[0146] The second index relationship writing module 9 is used to write the second index relationship into the key-value storage system.

[0147] Specifically, based on the idea of ​​inverted index, a term is used to quickly find the primary key set of all rows whose specific columns contain the term or the primary key set corresponding to the document.

[0148] Considering that for a table, each insertion of a row of data may modify the primary key set corresponding to any term in the inverted index, the inverted index needs to be modified frequently. Therefore, it is not suitable to be placed in a key-value storage system, but needs to be managed and stored separately. In this embodiment, Lucene is used to establish a secondary index relationship between the entry and the second primary key set to achieve the management and storage of the inverted index.

[0149] In this embodiment, when writing data to be written, the data to be written is first written into the key-value storage system in one transaction, and then the secondary index and inverted index are written, and the transaction is submitted to complete the data writing operation, thereby achieving the need to support relational query and full-text retrieval at the same time, reducing the implementation difficulty and complexity, and overcoming the existing defects of easily causing great waste of storage resources and network resources and easily leading to inaccurate full-text retrieval results.

[0150] Example 6

[0151] like Figure 5 As shown, the data writing system of this embodiment is a further improvement of embodiment 5, specifically:

[0152] The data writing system further includes a request sending module 10 , a serial number obtaining module 11 and a first writing module 12 .

[0153] The request sending module 11 is used to send a request to the key-value storage system to apply for writing the data to be written;

[0154] The serial number acquisition module 12 is used to generate a write serial number according to the request;

[0155] The first writing module 13 is used to write the write sequence number and the data to be written into the write-ahead log system.

[0156] The data writing system further includes a judgment module, which is used to judge whether the data to be written is successfully written into the key-value storage system. If it is successfully written, it continues to judge whether the data table has a secondary index column for relationship query. If it has a secondary index column, it calls the first index relationship establishment module 4 and continues to judge whether the data table has a full-text index column for full-text search. If it has a full-text index column, it calls the second primary key set acquisition module 7; or,

[0157] The judgment module is used to determine whether the data to be written is successfully written into the key-value storage system. If it is successfully written, it continues to determine whether there is a full-text index column for full-text retrieval in the data table. If there is a full-text index column, the second primary key set acquisition module 7 is called, and it continues to determine whether the data table has a secondary index column for relationship query. If there is a secondary index column, the first index relationship establishment module 4 is called.

[0158] In other words, in this embodiment, the data to be written is first ensured to be successfully written into the key-value storage system, and then the secondary index information and full-text index information are written separately, thereby enabling the key-value storage system to support both relational queries and full-text searches. The order in which the secondary index information and full-text index information are written is not specifically specified.

[0159] In addition, if a write operation to the key-value storage system fails, a rollback operation is performed and the information of the write failure is recorded.

[0160] In this embodiment, when writing data to be written, the data to be written is first written into the key-value storage system in one transaction, and then the secondary index and inverted index are written, and the transaction is submitted to complete the data writing operation, thereby achieving the need to support relational query and full-text retrieval at the same time, reducing the implementation difficulty and complexity, and overcoming the existing defects of easily causing great waste of storage resources and network resources and easily leading to inaccurate full-text retrieval results.

[0161] Example 7

[0162] The data query system of this embodiment is implemented by using the data writing system of any one of the fifth and sixth embodiments.

[0163] Specifically, if Figure 6 As shown, the data query system of this embodiment includes:

[0164] The data query system includes a filtering condition acquisition module 13 , a target primary key acquisition module 14 and a target data query module 15 .

[0165] The filtering condition acquisition module 13 is used to obtain the filtering condition;

[0166] The target primary key acquisition module 14 is configured to, when the filtering condition includes a value in a secondary index column and a target term, acquire a corresponding first primary key set based on the value in the secondary index column and a corresponding second primary key set based on the target term, and use the intersection of the first primary key set and the second primary key set as the target primary key set;

[0167] The target data query module 15 is used to query and obtain corresponding target data according to the target primary key set.

[0168] In this embodiment, the filtering conditions are combined and the primary key set that meets the conditions is queried based on the secondary index and the inverted index, and then the final target data is queried, thereby meeting the needs of supporting relational query and full-text retrieval at the same time, reducing the difficulty and complexity of implementation, and overcoming the existing defects of easily causing great waste of storage resources and network resources and easily leading to inaccurate full-text retrieval results.

[0169] Example 8

[0170] The data query system of this embodiment is a further improvement of the embodiment 7. Specifically:

[0171] The target primary key acquisition module 14 is used to directly acquire the target key value corresponding to the target primary key when the filtering condition only includes the target primary key, that is, no secondary indexing or full-text search is required;

[0172] The target data query module 15 is used to query the target data corresponding to the target primary key from the key value storage system according to the target key value. Or,

[0173] The target primary key acquisition module 14 is used to acquire the first primary key set corresponding to the values ​​in the secondary index column when the filtering condition only includes the values ​​in the secondary index column, that is, no full-text search is required;

[0174] The target data query module 15 is used to query and obtain the corresponding target data according to the first primary key set. Or,

[0175] The target data query module 15 is used to obtain the second primary key set corresponding to the target term when the filtering condition only includes the target term, that is, no secondary index is required;

[0176] The target data query module 15 is used to query and obtain corresponding target data according to the second primary key set.

[0177] In this embodiment, different filtering conditions are combined to obtain query results that match the filtering conditions, and queries on target data are independently implemented based on the target primary key, secondary index, and full-text index, thereby improving the existing data query method and enhancing the user data query experience.

[0178] Example 9

[0179] Figure 7 This is a schematic diagram of the structure of an electronic device provided in Example 9 of the present invention. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the data writing method in any one of Examples 1 or 2 is implemented. Figure 7 The electronic device 30 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present invention.

[0180] like Figure 7 As shown, the electronic device 30 may be a general-purpose computing device, such as a server device. Components of the electronic device 30 may include, but are not limited to, the at least one processor 31, the at least one memory 32, and a bus 33 connecting different system components (including the memory 32 and the processor 31).

[0181] The bus 33 includes a data bus, an address bus, and a control bus.

[0182] The memory 32 may include a volatile memory, such as a random access memory (RAM) 321 and / or a cache memory 322 , and may further include a read-only memory (ROM) 323 .

[0183] The memory 32 may also include a program / utility 325 having a set (at least one) of program modules 324, such program modules 324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0184] The processor 31 executes various functional applications and data processing by running the computer program stored in the memory 32, such as the data writing method in any one of Embodiments 1 or 2 of the present invention.

[0185] The electronic device 30 may also communicate with one or more external devices 34 (e.g., a keyboard, a pointing device, etc.). Such communication may be performed via an input / output (I / O) interface 35. Furthermore, the model generating device 30 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 36. Figure 7As shown, the network adapter 36 communicates with the other modules of the model-generated device 30 via the bus 33. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the model-generated device 30, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID (RAID) systems, tape drives, and data backup storage systems.

[0186] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above may be embodied in a single unit / module. Conversely, the features and functions of a single unit / module described above may be further divided and embodied by multiple units / modules.

[0187] Example 10

[0188] Embodiment 10 of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and run on the processor. When the processor executes the program, it implements the data query method in any one of Embodiments 3 or 4. The specific structure of the electronic device refers to the electronic device in Embodiment 9, and its working principle is basically the same as the working principle of the electronic device in Embodiment 9, which will not be repeated here.

[0189] Example 11

[0190] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the data writing method in any one of Embodiments 1 or 2 are implemented.

[0191] The readable storage medium may include, but is not limited to, a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0192] In a possible implementation, the present invention can also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to enable the terminal device to execute the steps in the data writing method in any one of Examples 1 or 2.

[0193] The program code for executing the present invention may be written in any combination of one or more programming languages, and may be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on the remote device.

[0194] Example 12

[0195] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the data query method in any one of Embodiments 3 or 4 are implemented.

[0196] The readable storage medium may include, but is not limited to, a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0197] In a possible implementation, the present invention can also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to enable the terminal device to execute the steps in the data query method in any one of Examples 3 or 4.

[0198] The program code for executing the present invention may be written in any combination of one or more programming languages, and may be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on the remote device.

[0199] Although specific embodiments of the present invention have been described above, those skilled in the art will appreciate that these are merely illustrative and that the scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, and such changes and modifications are intended to fall within the scope of the present invention.

Claims

1. A data writing method, characterized in that: The data writing method comprises: In one transaction, the data to be written is written into the key-value storage system in the form of key-value pairs; Wherein, the key-value storage system includes multiple data tables, and the key-value pairs include primary keys corresponding to the data tables; The format of the key is: table ID: primary key ID: primary key value; the value is a bit array converted from a string consisting of all column names and column values ​​in a row of data. Presetting a secondary index column in the data table for relational query; Obtain a first primary key set corresponding to the value of the secondary index column; Establishing a first index relationship between the value of the secondary index column and the first primary key set, and writing the first index relationship into the key-value storage system; Presetting a full-text index column in the data table for full-text search; Obtaining corresponding entries in the full-text index column, and obtaining second primary key sets corresponding to different data tables containing the entries; A second index relationship is established between the entry and the second primary key set, and the second index relationship is written into the key-value storage system.

2. The data writing method according to claim 1, wherein: Before the step of writing the data to be written into the key-value storage system in the form of a key-value pair, the following steps are also included: Sending a request to the key-value storage system to apply for writing the data to be written; generating a write sequence number according to the request; The write sequence number and the data to be written are written into a write-ahead log system.

3. The data writing method according to claim 1, wherein: After the step of writing the data to be written into the key-value storage system in the form of a key-value pair and before the step of obtaining the corresponding entry in the full-text index column, the following step is further included: Determine whether the data to be written is successfully written into the key-value storage system; if so, continue to determine whether the data table has a secondary index column for relationship query; if so, execute the steps of establishing a first index relationship between the value of the secondary index column and the first primary key set, and write the first index relationship into the key-value storage system; continue to determine whether the data table has a full-text index column for full-text search; if so, execute the step of obtaining the corresponding entry in the full-text index column; or, After the data to be written is successfully written into the key-value storage system, it is determined whether a full-text index column for full-text retrieval is provided in the data table; if so, the step of obtaining the corresponding entry in the full-text index column is executed, and it is further determined whether the data table is provided with a secondary index column for relational query; if so, the step of establishing a first index relationship between the value of the secondary index column and the first primary key set and writing the first index relationship into the key-value storage system is executed.

4. A data query method, characterized in that: The data query method is implemented by using the data writing method according to any one of claims 1 to 3, and the data query method includes: Get the filter conditions; When the filtering condition includes the value in the secondary index column and the target term, the corresponding first primary key set is obtained according to the value in the secondary index column and the corresponding second primary key set is obtained according to the target term; Taking the intersection of the first primary key set and the second primary key set as the target primary key set; The corresponding target data is obtained by querying the target primary key set.

5. The data query method according to claim 4, wherein: When the filtering condition only includes the target primary key, the target key value corresponding to the target primary key is obtained; Target data corresponding to the target primary key is obtained by querying from the key-value storage system according to the target key value.

6. The data query method according to claim 4, wherein: When the filtering condition only includes the values ​​in the secondary index column, obtaining the first primary key set corresponding to the values ​​in the secondary index column; The corresponding target data is obtained by querying according to the first primary key set.

7. The data query method according to claim 4, wherein: When the filtering condition includes only the target term, obtaining the second primary key set corresponding to the target term; The corresponding target data is obtained by querying the second primary key set.

8. A data writing system, characterized in that: The data writing system includes a data writing module, a secondary index column preset module, a first primary key set acquisition module, a first index relationship establishment module, a first index relationship writing module, a full-text index column preset module, a second primary key set acquisition module, a second index relationship establishment module, and a second index relationship writing module; The data writing module is used to write the data to be written into the key-value storage system in the form of key-value pairs in one transaction; Wherein, the key-value storage system includes multiple data tables, and the key-value pairs include primary keys corresponding to the data tables; The format of the key is: table ID: primary key ID: primary key value; the value is a bit array converted from a string consisting of all column names and column values ​​in a row of data. The secondary index column preset module is used to preset the secondary index column for relationship query in the data table; The first primary key set acquisition module is used to acquire a first primary key set corresponding to the value of the secondary index column; The first index relationship establishing module is used to establish a first index relationship between the value of the secondary index column and the first primary key set; The first index relationship writing module is used to write the first index relationship into the key-value storage system; The full-text index column preset module is used to preset the full-text index columns in the data table for full-text search; The second primary key set acquisition module is used to acquire the corresponding entry in the full-text index column, and acquire the second primary key set corresponding to the different data tables containing the entry; The second index relationship establishment module is used to establish a second index relationship between the entry and the second primary key set; The second index relationship writing module is used to write the second index relationship into the key-value storage system.

9. The data writing system according to claim 8, wherein: The data writing system further includes a request sending module, a serial number obtaining module and a first writing module; The request sending module is used to send a request to the key-value storage system to apply for writing the data to be written; The serial number acquisition module is used to generate a write serial number according to the request; The first writing module is used to write the write sequence number and the data to be written into a write-ahead log system.

10. The data writing system according to claim 8, wherein: The data writing system further includes a judgment module; The judgment module is used to judge whether the data to be written is successfully written into the key-value storage system. If it is successfully written, it continues to judge whether the data table is provided with a secondary index column for relationship query. If it is provided with the secondary index column, it calls the first index relationship establishment module and continues to judge whether the data table is provided with a full-text index column for full-text search. If it is provided with the full-text index column, it calls the second primary key set acquisition module; or, The judgment module is used to determine whether the data to be written is successfully written into the key-value storage system. If it is successfully written, it continues to determine whether a full-text index column for full-text retrieval is provided in the data table. If the full-text index column is provided, the second primary key set acquisition module is called, and the judgment module continues to determine whether the data table is provided with a secondary index column for relationship query. If the secondary index column is provided, the first index relationship establishment module is called.

11. A data query system, characterized in that: The data query system is implemented by the data writing system according to any one of claims 8 to 10, and the data query system includes a filtering condition acquisition module, a target primary key acquisition module and a target data query module; The filtering condition acquisition module is used to obtain filtering conditions; The target primary key acquisition module is configured to, when the filtering condition includes a value in the secondary index column and a target term, acquire the corresponding first primary key set according to the value in the secondary index column and acquire the corresponding second primary key set according to the target term, and use the intersection of the first primary key set and the second primary key set as the target primary key set; The target data query module is used to query and obtain corresponding target data according to the target primary key set.

12. The data query system according to claim 11, wherein: The target primary key acquisition module is used to acquire the target key value corresponding to the target primary key when the filtering condition only includes the target primary key; The target data query module is used to query the key-value storage system according to the target key value to obtain the target data corresponding to the target primary key.

13. The data query system according to claim 11, wherein: The target primary key acquisition module is configured to acquire the first primary key set corresponding to the values ​​in the secondary index column when the filtering condition only includes the values ​​in the secondary index column; The target data query module is used to query and obtain corresponding target data according to the first primary key set.

14. The data query system according to claim 11, wherein: The target data query module is configured to obtain the second primary key set corresponding to the target term when the filtering condition only includes the target term; The target data query module is used to query and obtain corresponding target data according to the second primary key set.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the data writing method according to any one of claims 1 to 3 is implemented.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the data query method according to any one of claims 4 to 7 is implemented.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data writing method according to any one of claims 1 to 3 are implemented.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data query method according to any one of claims 4 to 7 are implemented.

Images