A data query method and device

Abstract

The application provides a data query method and device, relates to the field of big data, and can also be used in the financial field, and comprises the following steps: receiving an anonymous query data request sent by a service channel client; obtaining corresponding queried data from an auxiliary database according to the anonymous query data request; wherein the queried data is the result of real-time backup of data in a background database in the auxiliary database; performing anonymous query processing on the queried data based on the anonymous query data request, and returning an anonymous query processing result to the service channel client. The application can decouple data-side application and channel-side application, and realizes data query under read-write separation based on an anonymous query algorithm.

Description

Technical Field

[0001] This application relates to the field of big data and can be used in the financial sector, specifically a data query method and device. Background Technology

[0002] In the current business environment, there are frequent interface calls between service channel clients (also known as service channel applications) and service channel clients (also known as service provider applications). Service channel applications call the query interface provided by the service provider applications to obtain the required data from the database of the service provider applications.

[0003] Figure 1 and Figure 2 Two existing technical solutions for implementing the above process are shown. The first solution is the simplest and most direct: the service channel requests data from the main data source through the query interface provided by the service provider application, and the service provider application's database returns the query results. This solution does not perform any decoupling or load reduction; the service channel application and the service provider application access the same database. The second solution is a relatively advanced read-write separation solution for the service provider application. The service provider application separates its database for read and write operations, providing separate write and read databases. Although this solution reduces the load on the service provider application's main database by transferring some read pressure to the read database, the service channel application and the service provider application are still not decoupled; the service channel application still needs to access the service provider application's database. Summary of the Invention

[0004] To address the problems in the prior art, this application provides a data query method and apparatus that can decouple data-side applications from channel-side applications and achieve data querying under read-write separation based on a hidden query algorithm.

[0005] To solve the above-mentioned technical problems, this application provides the following technical solution:

[0006] Firstly, this application provides a data query method, including:

[0007] Receive hidden query data requests sent by the service channel client;

[0008] The corresponding queried data is obtained from the auxiliary database according to the hidden query data request; wherein, the queried data is the result of a near real-time backup of the data in the background database in the auxiliary database;

[0009] Based on the hidden query data request, the queried data is processed using a hidden query, and the result of the hidden query is returned to the service channel client.

[0010] Further, the hidden query data request includes the user sub-identifier group generated by the service channel client; the step of obtaining the corresponding queried data from the auxiliary database according to the hidden query data request includes:

[0011] Based on the hidden query data request, determine whether the queried data corresponding to the identifier in the user sub-identifier group belongs to hot data;

[0012] If so, the corresponding hot data is read from the distributed cache of the auxiliary database as the queried data; wherein, the hot data is periodically stored in the distributed cache;

[0013] If not, read non-hotspot data from the unbuffered storage of the auxiliary database as the queried data.

[0014] Further, the step of performing hidden query processing on the queried data based on the hidden query data request, and returning the hidden query processing result to the service channel client, includes:

[0015] A hidden query calculation is performed on each identifier in the user sub-identifier group and the queried data to obtain the calculation result corresponding to each identifier;

[0016] According to the order of each identifier in the user sub-identifier group, the calculation results are combined into a calculation result group and returned to the service channel client as the hidden query processing result.

[0017] Furthermore, the data query method further includes:

[0018] The service receives data write requests sent by the client;

[0019] Write the corresponding data to the backend database according to the data write request;

[0020] The data in the background database is backed up to the auxiliary database in near real-time.

[0021] Furthermore, the near real-time backup of data from the background database to the auxiliary database includes:

[0022] The host replication interface is invoked in near real-time to perform data replication operations on the host database in the background database;

[0023] According to the data application requirements, the copied host database data is saved to the auxiliary database.

[0024] Furthermore, the near real-time backup of data from the background database to the auxiliary database includes:

[0025] The platform database replication interface is invoked in near real-time to perform data replication operations on the platform database in the background database;

[0026] According to the data application requirements, the copied platform database data is saved to the auxiliary database.

[0027] Further, the step of performing a hidden query calculation on each identifier in the user sub-identifier group and the queried data to obtain the calculation result corresponding to each identifier includes:

[0028] Obtain the full data under the query channel corresponding to the user sub-identifier group, as the queried data; the full data includes identifiers and information;

[0029] A user identifier score list is established based on the full data;

[0030] Based on the user identifier score list, a hidden query calculation is performed on the queried data to obtain the calculation results corresponding to each identifier.

[0031] Secondly, this application provides a data query device, comprising:

[0032] The query request receiving unit is used to receive hidden query data requests sent by the service channel client;

[0033] The query data reading unit is used to obtain the corresponding queried data from the auxiliary database according to the hidden query data request; wherein, the queried data is the result of a near real-time backup of the data in the background database in the auxiliary database;

[0034] The hidden query processing unit is used to perform hidden query processing on the queried data based on the hidden query data request, and return the hidden query processing result to the service channel client.

[0035] Furthermore, the concealed query data request includes a user sub-identifier group generated by the service channel client; the query data reading unit includes:

[0036] The hot data determination module is used to determine whether the queried data corresponding to the identifier in the user sub-identifier group belongs to hot data based on the hidden query data request.

[0037] The hot data reading module is used to read the corresponding hot data from the distributed cache of the auxiliary database as the queried data when it is determined whether the data corresponding to the identifier in the user sub-identifier group belongs to hot data; wherein, the hot data is periodically stored in the distributed cache;

[0038] The non-hotspot data reading module is used to read non-hotspot data from the non-buffered storage of the auxiliary database as the queried data when it is determined whether the data corresponding to the identifier in the user sub-identifier group is not hotspot data.

[0039] Furthermore, the hidden query processing unit includes:

[0040] The calculation result generation module is used to perform hidden query calculations on each identifier in the user sub-identifier group and the queried data to obtain the calculation results corresponding to each identifier.

[0041] The result order return module is used to group the calculation results into a calculation result group according to the order of each identifier in the user sub-identifier group, and return it to the service channel client as the hidden query processing result.

[0042] Furthermore, the data query device further includes:

[0043] The write request receiving unit is used to receive data write requests sent by the service provider's client;

[0044] A data writing unit is used to write corresponding data to the background database according to the data writing request.

[0045] The data backup unit is used to back up the data in the background database to the auxiliary database in near real-time.

[0046] Furthermore, the data backup unit includes:

[0047] The host data replication module is used to call the host replication interface in near real-time to perform data replication operations on the host database in the background database;

[0048] The host data storage module is used to save the copied host database data to the auxiliary database according to the data application requirements.

[0049] Furthermore, the data backup unit includes:

[0050] The platform data replication module is used to call the platform database replication interface in near real-time to perform data replication operations on the platform database in the background database;

[0051] The platform data storage module is used to save the copied platform database data to the auxiliary database according to the data application requirements.

[0052] Furthermore, the calculation result generation module includes:

[0053] The full data acquisition module is used to acquire the full data under the query channel corresponding to the user sub-identifier group as the queried data; the full data includes identifiers and information;

[0054] The score list creation module is used to create a user identifier score list based on the full data.

[0055] The calculation result generation module is used to perform hidden query calculations on the queried data based on the user identifier score list to obtain the calculation results corresponding to each identifier.

[0056] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the data query method.

[0057] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data query method.

[0058] Fifthly, this application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the data query method.

[0059] To address the problems in the prior art, the data query method and apparatus provided in this application can decouple the data access process between the service provider application and the service channel application, taking into account the characteristics of data query and retrieval in service channel applications. This overcomes the shortcomings of the prior art, such as low data protection security, high cost, low efficiency, and strong coupling, and realizes covert query between the service channel application and the main data source. This protects the privacy requirements of the service channel application when performing data queries, while reducing the access pressure on the database of the service provider application. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is one of the existing application connection methods;

[0062] Figure 2 This is the second application connection method in the existing technology;

[0063] Figure 3This is one of the architectural diagrams of the data query system in the embodiments of this application;

[0064] Figure 4 This is the second schematic diagram of the architecture of the data query system in the embodiments of this application;

[0065] Figure 5 This is one of the schematic diagrams of hidden query calculation in the embodiments of this application;

[0066] Figure 6 This is the second schematic diagram of the hidden query calculation in the embodiments of this application;

[0067] Figure 7 This is one of the flowcharts for the data query method in the embodiments of this application;

[0068] Figure 8 This is a flowchart illustrating the process of reading the corresponding queried data in this application embodiment;

[0069] Figure 9 This is a flowchart illustrating the hidden query processing in this application embodiment;

[0070] Figure 10 This is the second flowchart of the data query method in the embodiments of this application;

[0071] Figure 11 This is one of the flowcharts for data backup in the embodiments of this application;

[0072] Figure 12 This is the second flowchart of data backup in the embodiments of this application;

[0073] Figure 13 This is a flowchart illustrating the hidden query calculation in the embodiments of this application;

[0074] Figure 14 This is one of the structural diagrams of the data query device in the embodiments of this application;

[0075] Figure 15 This is a structural diagram of the data reading unit in an embodiment of this application;

[0076] Figure 16 This is a structural diagram of the hidden query processing unit in the embodiments of this application;

[0077] Figure 17 This is the second structural diagram of the data query device in the embodiments of this application;

[0078] Figure 18 This is one of the structural diagrams of the data backup unit in the embodiments of this application;

[0079] Figure 19 This is the second structural diagram of the data backup unit in the embodiments of this application;

[0080] Figure 20 This is a structural diagram of the calculation result generation module in the embodiments of this application;

[0081] Figure 21 This is a schematic diagram of the structure of the electronic device in the embodiments of this application;

[0082] Figure 22 This is a schematic diagram of the data copying structure in an embodiment of this application. Detailed Implementation

[0083] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0084] It should be noted that the data query method and apparatus provided in this application can be used in the financial field, or in any field other than the financial field. The application field of the data query method and apparatus provided in this application is not limited.

[0085] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0086] In one embodiment, see Figure 7 In order to decouple data-side applications from channel-side applications and implement data querying under read-write separation based on a hidden query algorithm, this application provides a data querying method, including:

[0087] S101: Receive a hidden query data request sent by the service channel application (also known as the service channel client);

[0088] S102: Obtain the corresponding queried data from the auxiliary database according to the hidden query data request; wherein, the queried data is the result of a near real-time backup of the data in the background database in the auxiliary database;

[0089] S103: Based on the hidden query data request, perform hidden query processing on the queried data, and return the hidden query processing result to the service channel application.

[0090] Understandably, to overcome the shortcomings of existing technologies such as low data protection security, high cost, low efficiency, and strong coupling between applications, and to achieve a hidden query, decoupled, and efficient data query service, this application provides a hidden query-based data query method. This method, based on a hidden query algorithm, efficient near real-time data query technology, and application decoupling technology, copies relevant table data from the databases of various service-providing applications (also known as service-providing clients) to an auxiliary database. It then provides a unified and complete hidden query online interface according to the needs of each service channel application, offering near real-time data query services tailored to business scenarios, and providing shared query and information integration services for various channels. The execution entity for steps S101 to S103 is a server or a terminal, which carries the hidden query online interface and a proxy layer. The hidden query online interface and proxy layer are also referred to as the hidden query auxiliary system.

[0091] The following is a brief explanation of each term.

[0092] Service channel applications: Applications that provide certain services to customers, such as credit card service channel applications, which can provide services such as credit card application.

[0093] Service-providing applications: These are applications that provide specific data to service channel applications when providing services to external parties. Their databases store certain data required by the service channel applications. For example, a customer information service-providing application might store customer information, including but not limited to ID numbers, required when a customer applies for a credit card.

[0094] Hotspot data: Data on the number of times service channel applications are accessed within a unit of time.

[0095] Private Information Retrieval (PIR) refers to a query method in which the data queryer (a service channel application in this embodiment) obtains the data they need without disclosing their query intent to the data provider (a service provider application in this embodiment), and without the data queryer having access to other data in the data provider's database.

[0096] See Figure 3 As shown below, the system architecture of the embodiments of this application will be described in detail.

[0097] Main data source 101: Main data source 101 is the collection of data (databases) of service provider application 102 that service channel application 108 needs to query. Main data source 101 is also called the backend database, including: host database and platform database. The host database refers to the IBM mainframe DB2 database, and the platform database refers to common databases such as Oracle, MySQL, and GaussDB. Whether the main data source is stored in the host database or the platform database depends on which database the specific business runs on; this application embodiment does not limit this. This application embodiment can support multiple mainstream databases, including the above-mentioned types of databases.

[0098] Service Provider Application 102: The relevant data required by Service Channel Application 108 when providing services ultimately comes from Service Provider Application 102, which provides the source data for near real-time data replication operation 103.

[0099] Near real-time data replication operation 103: Provides near real-time data replication services at the second level, including replication between the host database and the auxiliary database, and replication between the platform database and the auxiliary database.

[0100] Hidden query data source 104: also known as the auxiliary database. When the service channel application 108 provides services to external parties, the hidden query data source 104 provides data to the service channel application 108 through a concealment algorithm. It is the target database of the near real-time data replication 103.

[0101] Hidden query auxiliary system 105: A system device that directly provides hidden data query for service channel application 108, consisting of hidden query data source 104, online interface and proxy layer 106, and can implement hidden query algorithm.

[0102] Online interface and proxy layer 106: Provides an interface for data query services to service channel application 108 and performs certain data processing to speed up access. The database it queries is the hidden query data source 104, which is the auxiliary database.

[0103] 107 Service Channel: This is a collection of service channel applications 108 that provide raw data from the main data source 101 to serve customers. It uses a hidden query algorithm to query the required data from the hidden query auxiliary system 105.

[0104] Service Channel Application 108: A channel scenario that provides services to customers. When providing services, it needs relevant data from Service Provider Application 102, and the data provided directly to it is the auxiliary database of Hidden Query Assistance System 105.

[0105] As can be seen from the above description, the data query method provided in this application can decouple the data access process between the service provider application and the service channel application, taking into account the characteristics of data query and reading in the service channel application. This overcomes the shortcomings of the prior art, such as low data protection security, high cost, low efficiency, and strong coupling, and realizes hidden query between the service channel application and the main data source. This protects the privacy requirements of the service channel application when performing data queries, while reducing the access pressure on the database of the service provider application.

[0106] The following provides a detailed explanation of steps S101 to S103.

[0107] Step S101: Receive the hidden query data request sent by the service channel application.

[0108] It is understandable that the execution entity of step S101 is a server or terminal equipped with an online interface and proxy layer 106, which receives a hidden query data request sent by the service channel application. Specifically, the service channel application initiates a hidden query data request based on actual business needs, enabling the online interface and proxy layer 106 to assist in completing the data query in a hidden manner, ultimately obtaining the data corresponding to the actual business.

[0109] As can be seen from the above description, the data query method provided in this application is capable of receiving hidden data query requests sent by service channel applications.

[0110] Step S102: Obtain the corresponding queried data from the auxiliary database according to the hidden query data request; wherein, the queried data is the result of a near real-time backup of the data in the background database in the auxiliary database.

[0111] It is understood that the execution entity of step S101 is still the server or terminal equipped with the online interface and proxy layer 106, which retrieves the corresponding queried data from the auxiliary database according to the hidden query data request. When the online interface and proxy layer 106 receive the hidden query data request, it needs to query the data required by the service channel application in the hidden query data source 104 (i.e., the auxiliary database) using a hidden query method, based on the relevant information recorded in the hidden query data request. To demonstrate the application decoupling in this embodiment, the queried data has been pre-copied from the backend database to the auxiliary database, and the copying process is near real-time.

[0112] Figure 8 This is a specific embodiment of the data query method implemented in this application.

[0113] In one embodiment, see Figure 8The hidden query data request includes the user sub-identifier group generated by the service channel application; the step of obtaining the corresponding queried data from the auxiliary database according to the hidden query data request includes:

[0114] S201: Determine whether the data being queried corresponding to the identifier in the user sub-identifier group belongs to hot data based on the hidden query data request;

[0115] S202: If so, read the corresponding hot data from the distributed cache of the auxiliary database as the queried data; wherein, the hot data is periodically stored in the distributed cache;

[0116] S203: If not, read non-hotspot data from the unbuffered storage of the auxiliary database as the queried data.

[0117] It is understood that in this embodiment, the storage entity of the auxiliary database includes a distributed cache and an unbuffered memory. To achieve efficient access to the auxiliary database, hot data can be added to the distributed memory cache, and the consistency between the cache and the main data source 101 can be periodically ensured, thereby improving the access efficiency of hot data and reducing the access pressure on the main data source 101.

[0118] It should also be noted that the anonymous query data request includes a user sub-identifier group generated by service channel application 108. This user sub-identifier group is essential in the anonymous query process. This group contains both the identifier of the user to be queried and the identifiers of other users used to achieve privacy obfuscation. This user sub-identifier should be predefined within the system, enabling both parties involved in the query to obtain the corresponding query channel within a certain range. After receiving the user sub-identifier group, the online interface and proxy layer 106 can provide the full data query results for that channel, which will then be used as the queried data.

[0119] As can be seen from the above description, the data query method provided in this application can obtain the corresponding queried data from the auxiliary database according to the hidden query data request.

[0120] Step S103: Perform hidden query processing on the queried data based on the hidden query data request, and return the hidden query processing result to the service channel application.

[0121] Understandably, the execution entity of step S103 is still the server or terminal equipped with the online interface and proxy layer 106. It performs hidden query processing on the queried data based on the hidden query data request and returns the hidden query processing result to the service channel application. Since the queried data has been determined in the previous step, and the queried data is the full data under this query channel, in order to ensure that other data in the data provider's database is not obtained by the data querying party, it is necessary to perform hidden query processing on the queried data before returning the hidden query processing result to the service channel application.

[0122] Figure 9 This is a specific embodiment of the data query method implemented in this application.

[0123] In one embodiment, see Figure 9 The step of performing hidden query processing on the queried data based on the hidden query data request and returning the hidden query processing result to the service channel application includes:

[0124] S301: Perform a hidden query calculation on each identifier in the user sub-identifier group and the queried data to obtain the calculation result corresponding to each identifier;

[0125] S302: According to the order of each identifier in the user sub-identifier group, the calculation results are combined into a calculation result group and returned to the service channel application as the hidden query processing result.

[0126] It is understood that, in one embodiment, see [reference] Figure 13 The step of performing a hidden query calculation on each identifier in the user sub-identifier group and the queried data to obtain the calculation result corresponding to each identifier includes:

[0127] S701: Obtain the full data under the query channel corresponding to the user sub-identifier group as the queried data; the full data includes identifiers and information;

[0128] S702: Establish a user identifier score list based on the full data;

[0129] S703: Perform a hidden query calculation on the queried data based on the user identifier score list to obtain the calculation results corresponding to each identifier.

[0130] In one embodiment, see Figure 4 The hidden query process is as follows:

[0131] ① Before sending its anonymous query data request, service channel application 108 first merges the identifier of the queried user with the identifiers of multiple users randomly selected from the user database to form a user group {A1, A2, A3}. Each user identifier in the user group is separated into N sub-identifiers through key sharing. In this embodiment, N=3 is taken as an example, that is, A1 is further divided into {A11, A12, A13} to ensure that A1 = A11 + A12 + A13, forming a user sub-identifier group {A11, A12, A13, A21, A22, A23, A31, A32, A33}, which is then sent to the anonymous query online interface and proxy layer 106. See [link to documentation]. Figure 5 As shown.

[0132] ② The hidden query online interface and proxy layer 106, based on the previously agreed user sub-identifier group, retrieves the full data under the corresponding query channel of that sub-identifier group. This full data includes user identifier and user information [KEY,VALUE]. The user identifier and user information are divided into a user identifier score list: {[B1,v1],[B2,v2],[B3,v3],[B4,v4]}. See [link / reference] Figure 6 As shown. The KEY is a preset key that provides encryption parameters for the actual VALUE value, used to obfuscate the VALUE value during the hidden query calculation process.

[0133] ③ The hidden query online interface and proxy layer 106 receive the user sub-identifier group sent by the service channel application 108, and performs a hidden query calculation on each user sub-identifier and each item {B1, B2, B3, B4} in its own user identifier score list. After receiving any sub-digit identifier in {A11, A12, A13}, it can perform corresponding calculations on each digit identifier in {B1, B2, B3, B4} according to the classification information of the sub-digit identifier. The classification can be based on the second subscript, where an odd second subscript is classified as the first category, and an even second subscript as the second category. Therefore, if the second subscript of A11 and A13 is odd, subtraction is applied; if the second subscript of A12 is even, addition is applied. Taking B1 as an example, the calculation method is based on A11 and B1, A12 and B1, and A13 and B1, where the second subscript of the identifier is odd for subtraction and even for addition.

[0134] Calculation method for A11 and B1: D111 = A11 - B1, C111 = v1 A11-B1

[0135] Calculation method for A12 and B1: D121 = A12 + B1, C121 = v1 A12+B1

[0136] Calculation method for A13 and B1: D131 = A13 - B1, C131 = v1 A13-B1

[0137] Form the calculation result set {D,C}:

[0138] {[D111,C111],[D121,C121],[D131,C131],...,[Di,Ci],...[Dn,Cn]}

[0139] The calculation results are returned in the order of the identifiers in the user sub-identifier group and sent back to the service channel application 108.

[0140] ④Receive the calculation result group {D,C} from channel 209, and extract {Di,Ci} corresponding to A11, A12, and A13:

[0141] {[D111,C111],[D121,C121],[D131,C131],...,[D11i,C11i],[D12i,C12i],[D13i,C13i],...,[D11n,C11n],[D11n,C12n],[D11n,C13n]},

[0142] Extract the calculation results where D11i+D12i+D13i is 0.

[0143] Because D11i+D12i+D13i

[0144] = (A11-Bi)+(A12+Bi)+(A13-Bi)

[0145] =A11+A12+A13-Bi

[0146] =A1-Bi

[0147] Therefore, D11i+D12i+D13i is 0, which means that Bi = A1.

[0148] Service channel application 108 obtains the corresponding calculation result group: [D11i,C11i],[D12i,C12i],[D13i,C13i]. Since A1 = Bi, v1 is obtained through secret sharing calculation. Calculation method:

[0149] C11i×C12i×C13i

[0150] =v1 A11-Bi ×v1 A12+Bi ×v1 A13-Bi

[0151] =v1 A11-Bi+A12+Bi+A13-Bi

[0152] =v1 A11+A12+A13-Bi+Bi-Bi

[0153] =v1 A1-Bi

[0154] =v1

[0155] Service channel application 108 ultimately obtained the corresponding user information Vl, which is the query result corresponding to the hidden query data request.

[0156] It should be noted that the specific operations involved in the hidden query algorithm can be flexibly set according to the application scenario. The calculation process in this embodiment is only an example.

[0157] As can be seen from the above description, the data query method provided in this application can perform hidden query processing on the queried data based on the hidden query data request, and return the hidden query processing result to the service channel application.

[0158] Figure 10 This is a specific embodiment of the data query method implemented in this application.

[0159] In one embodiment, see Figure 10 The data query method further includes:

[0160] S401: Receive a data write request sent by the service provider application;

[0161] S402: Write the corresponding data to the background database according to the data write request;

[0162] S403: Back up the data in the background database to the auxiliary database in near real-time.

[0163] It is understandable that the server or terminal equipped with the online interface and the proxy layer 106 is also equipped with a data writing interface and the proxy layer 109, which can receive data writing requests sent by the service providing application and write the corresponding data to the background database according to the data writing request.

[0164] It should be noted that the embodiments of this application achieve read-write separation based on application decoupling. Although the online interface and proxy layer 106 and the data writing interface and proxy layer 109 are both mounted on the same server or terminal, their functions are independent of each other. The former is responsible for data querying, and the latter is responsible for data writing. The two can be performed simultaneously and independently.

[0165] The application scenario for writing data can be as follows: A service-providing application performs data maintenance according to business needs, thereby initiating a data write request to a server or terminal equipped with a data write interface and proxy layer 109. The server or terminal with the data write interface and proxy layer 109 receives and responds to the data write request to complete the data write operation. After the data is written to the backend database, the data in the backend database can be backed up to the auxiliary database in near real-time. The data stored in the auxiliary database can be used to provide data to the online interface and proxy layer 106 for subsequent data querying.

[0166] Step S403: Back up the data in the background database to the auxiliary database in near real-time.

[0167] Figure 11 This is a specific embodiment of the data query method implemented in this application.

[0168] In one embodiment, see Figure 11 The near real-time backup of data from the background database to the auxiliary database includes:

[0169] S501: In near real-time, call the host replication interface to perform a data replication operation on the host database in the background database;

[0170] S502: Save the copied host database data to the auxiliary database according to the data application requirements.

[0171] It should be noted that the host replication process is described in detail below. Figure 22 As shown, the topmost link, IBM QREP-DB2 parsing, sends data to the distributed message middleware Kakfa, and then the unified data write program completes the writing to the auxiliary databases (including Oracle, MySQL, and other databases). IBM QREP (IBM Data Replication) is a replication software; after IBM QREP replicates the data, the DB2 parsing tool performs data parsing, and the unified data write program completes the data write operation.

[0172] Step S403: Back up the data in the background database to the auxiliary database in near real-time.

[0173] Figure 12 This is a specific embodiment of the data query method implemented in this application.

[0174] In one embodiment, see Figure 12 The near real-time backup of data from the background database to the auxiliary database includes:

[0175] S601: In near real-time, call the platform database replication interface to perform a data replication operation on the platform database in the background database;

[0176] S602: Save the copied platform database data to the auxiliary database according to the data application requirements.

[0177] It is understandable that the execution flow of steps S501 to S502 is the same as that of steps S601 and S602, the only difference being whether the main data source comes from the host database or the platform database.

[0178] See Figure 22 As shown, IBM CDC is IBM's data replication product. Existing database replication products can also replicate data from platforms such as Oracle, MySQL, GaussDB, and OceanBase, sending the replicated data to Kafka and then using a unified write tool to complete the writing to the auxiliary database. These database replication products include, but are not limited to, Data Replication Service (DRS) and OceanBase Migration Service (OMS).

[0179] In application, see Figure 4 This can be understood by following the process below.

[0180] like Figure 4 As shown, the general data processing flow of the present invention is described in detail below (taking a single service-providing application as an example):

[0181] Service Provider Application 102: This process takes a single service provider application as an example. The service provider application may include host DB2 and platform database, which store raw data information and are the source of shared data.

[0182] Host DB2: Host application database, which is the source database for host replication.

[0183] Platform Database: The platform application database is the source database for platform replication. Some applications may have both host DB2 and platform databases coexisting.

[0184] Master Replication: Near real-time replication from the master database to the platform database, achieved using a master replication tool. The source database is the master database DB2, and the target database is the auxiliary database. The data flow is from the master database DB2 to the master replication tool and then to the auxiliary database.

[0185] Platform replication: Near real-time platform-to-platform database replication at the second level, achieved using a platform database replication tool. The source database is the platform database, and the target database is the auxiliary system platform database. The data flow is from the platform database to the platform replication (tool) and then to the auxiliary system platform database.

[0186] The hidden query assistance system includes an auxiliary database, an online interface, and a proxy layer. The internal data flow is from the auxiliary database to the online interface and the proxy layer.

[0187] Auxiliary database: Depending on different needs and data replication scenarios, different databases or database clusters such as Oracle and MySQL may be involved. As the target database for host replication and platform replication, it receives source data synchronized from the service provider application side by host replication (tool) and platform replication (tool).

[0188] Hidden Query Online Interface and Proxy Layer: The service channel application initiates a hidden query data request to the online interface and proxy layer. The online interface and proxy layer read data from the auxiliary database and returns it to the service channel application through the hidden query algorithm. At the same time, the proxy layer will perform certain speed-up processing on the data to speed up data access.

[0189] Service Channel Applications: A service provider application may provide data to multiple service channel applications, which are the final destinations for shared data. The service channel application acts as the data queryer for hidden queries, while the online interface and proxy layer act as the data provider for hidden queries, completing the hidden query business.

[0190] In summary, this invention provides an efficient and cost-effective application decoupling method and apparatus that enables service channels to perform hidden queries on the main data source, achieves zero access to the native database without intrusion, and has at least the following advantages:

[0191] Beneficial effects:

[0192] 1. Secure Data Access:

[0193] Through covert query technology, service channel applications, acting as the querying party, can conceal their query intent, protecting user privacy and ensuring that shared data is not directly read between different applications. Simultaneously, service provider applications can provide query information as needed based on their own business requirements. Furthermore, after the queried data is processed using the covert query algorithm, the transmitted data remains unreadable encrypted, preventing non-querying parties and data providers from performing calculations, thus protecting the confidentiality of the transmission process.

[0194] 2. Efficient application decoupling:

[0195] 1) High-efficiency read-write separation between host and platform: Through a near real-time data replication tool, the host DB2 and the platform database achieve high-efficiency near real-time data synchronization at the second level, realizing high-efficiency and high-accuracy read-write separation between host and platform.

[0196] 2) Efficient access to the auxiliary system platform database: By adding hot data to the distributed memory cache, the consistency between the cache and the database is periodically guaranteed, improving the access efficiency of hot data and reducing the access pressure on the database.

[0197] 3. Achieve zero access and non-intrusion to the main data source database by service channel applications: Service channel applications need to obtain data from service provider applications, but they do not need to care about the actual location of service provider applications or access the database of service provider applications. Instead, the hidden query assistance system directly provides the required data to service channel applications, thereby reducing the pressure on the database of service provider applications.

[0198] 4. Low cost: Read and write separation is achieved between the host and the platform. The platform database provides all read services to the outside world instead of the host DB2, effectively reducing the pressure on the host DB2 and saving expensive host resources.

[0199] 5. Application decoupling: Data-side applications and channel-side applications are independent of each other and do not need to be aware of each other's existence. This allows for data sharing without affecting each other's services, and the normality or non-normality of one party's service will not affect the other party's service.

[0200] Based on the same inventive concept, this application also provides a data query device, which can be used to implement the method described in the above embodiments, as described in the following embodiments. Since the principle of the data query device in solving the problem is similar to that of the data query method, the implementation of the data query device can refer to the implementation of the method based on software performance benchmarks, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0201] In one embodiment, see Figure 14 To decouple data-side applications from channel-side applications and implement data querying under read-write separation based on a hidden query algorithm, this application provides a data query device, comprising:

[0202] The query request receiving unit 1301 is used to receive hidden query data requests sent by the service channel application;

[0203] The query data reading unit 1302 is used to obtain the corresponding queried data from the auxiliary database according to the hidden query data request; wherein, the queried data is the result of a near real-time backup of the data in the background database in the auxiliary database;

[0204] The hidden query processing unit 1303 is used to perform hidden query processing on the queried data based on the hidden query data request, and return the hidden query processing result to the service channel application.

[0205] In one embodiment, see Figure 15 The hidden query data request includes the user sub-identifier group generated by the service channel application; the query data reading unit 1302 includes:

[0206] Hotspot data determination module 1401 is used to determine whether the queried data corresponding to the identifier in the user sub-identifier group belongs to hotspot data based on the hidden query data request.

[0207] The hot data reading module 1402 is used to read the corresponding hot data from the distributed cache of the auxiliary database as the queried data when it is determined whether the data corresponding to the identifier in the user sub-identifier group belongs to hot data; wherein, the hot data is periodically stored in the distributed cache.

[0208] The non-hotspot data reading module 1403 is used to read non-hotspot data from the non-buffered storage of the auxiliary database as the queried data when it is determined whether the data corresponding to the identifier in the user sub-identifier group is not hotspot data.

[0209] In one embodiment, see Figure 16 The hidden query processing unit 1303 includes:

[0210] The calculation result generation module 1501 is used to perform hidden query calculations on each identifier in the user sub-identifier group and the queried data to obtain the calculation results corresponding to each identifier.

[0211] The result order return module 1502 is used to group the calculation results into a calculation result group according to the order of each identifier in the user sub-identifier group, and return it to the service channel application as the hidden query processing result.

[0212] In one embodiment, see Figure 17 The data query device further includes:

[0213] The write request receiving unit 1601 is used to receive data write requests sent by the service providing application;

[0214] The data writing unit 1602 is used to write corresponding data to the background database according to the data writing request.

[0215] The data backup unit 1603 is used to back up the data in the background database to the auxiliary database in near real-time.

[0216] In one embodiment, see Figure 18 The data backup unit 1603 includes:

[0217] The host data replication module 1701 is used to call the host replication interface in near real-time to perform data replication operations on the host database in the background database;

[0218] The host data storage module 1702 is used to save the copied host database data to the auxiliary database according to the data application requirements.

[0219] In one embodiment, see Figure 19 The data backup unit 1603 includes:

[0220] The platform data replication module 1801 is used to call the platform database replication interface in near real-time to perform data replication operations on the platform database in the background database;

[0221] The platform data storage module 1802 is used to save the copied platform database data to the auxiliary database according to the data application requirements.

[0222] In one embodiment, see Figure 20 The calculation result generation module 1501 includes:

[0223] The full data acquisition module 2001 is used to acquire the full data under the query channel corresponding to the user sub-identifier group as the queried data; the full data includes identifiers and information;

[0224] The score list creation module 2002 is used to create a user identifier score list based on the full data.

[0225] The calculation result generation module 2003 is used to perform hidden query calculations on the queried data based on the user identifier score list to obtain the calculation results corresponding to each identifier.

[0226] From a hardware perspective, in order to decouple data-side applications from channel-side applications and implement data querying under read-write separation based on a hidden query algorithm, this application provides an embodiment of an electronic device for implementing all or part of the data querying method. The electronic device specifically includes the following components:

[0227] The system comprises a processor, a memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between the data query device and core business systems, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the embodiments of the data query method and the data query device in the embodiments, the content of which is incorporated herein, and repeated details will not be described again.

[0228] It is understood that the user terminal may include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc. Among these, the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0229] In practical applications, the data query method can be partially executed on the electronic device side as described above, or all operations can be completed on the client device. The choice can be made based on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are completed on the client device, the client device may further include a processor.

[0230] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0231] Figure 21 This is a schematic block diagram illustrating the system configuration of the electronic device 9600 according to an embodiment of this application. Figure 21 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that... Figure 21 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0232] In one embodiment, the data query method functionality can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0233] S101: Receive a hidden query data request sent by the service channel application;

[0234] S102: Obtain the corresponding queried data from the auxiliary database according to the hidden query data request; wherein, the queried data is the result of a near real-time backup of the data in the background database in the auxiliary database;

[0235] S103: Based on the hidden query data request, perform hidden query processing on the queried data, and return the hidden query processing result to the service channel application.

[0236] As can be seen from the above description, the data query method and apparatus provided in this application can decouple the data access process between the service provider application and the service channel application, taking into account the characteristics of data query and reading in the service channel application. This overcomes the shortcomings of the prior art, such as low data protection security, high cost, low efficiency, and strong coupling, and realizes covert query between the service channel application and the main data source. This protects the privacy requirements of the service channel application when performing data queries, while reducing the access pressure on the database of the service provider application.

[0237] In another embodiment, the data query device can be configured separately from the central processing unit 9100. For example, the data query device can be configured as a chip connected to the central processing unit 9100, and the data query method function can be implemented through the control of the central processing unit.

[0238] like Figure 21 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 21 All components shown; in addition, the electronic device 9600 may also include Figure 21 For components not shown, please refer to existing technologies.

[0239] like Figure 21 As shown, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives inputs and controls the operation of various components of the electronic device 9600.

[0240] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0241] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0242] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0243] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device's communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0244] The communication module 9110 is a transmitter / receiver 9110 that transmits and receives signals via the antenna 9111. The communication module (transmitter / receiver) 9110 is coupled to the central processing unit 9100 to provide input signals and receive output signals, which can be the same as in a conventional mobile communication terminal.

[0245] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is also coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored sound via the speaker 9131.

[0246] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the data query method with the execution subject being a server or client in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the data query method with the execution subject being a server or client in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0247] S101: Receive a hidden query data request sent by the service channel application;

[0248] S102: Obtain the corresponding queried data from the auxiliary database according to the hidden query data request; wherein, the queried data is the result of a near real-time backup of the data in the background database in the auxiliary database;

[0249] S103: Based on the hidden query data request, perform hidden query processing on the queried data, and return the hidden query processing result to the service channel application.

[0250] As can be seen from the above description, the data query method and apparatus provided in this application can decouple the data access process between the service provider application and the service channel application, taking into account the characteristics of data query and reading in the service channel application. This overcomes the shortcomings of the prior art, such as low data protection security, high cost, low efficiency, and strong coupling, and realizes covert query between the service channel application and the main data source. This protects the privacy requirements of the service channel application when performing data queries, while reducing the access pressure on the database of the service provider application.

[0251] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0252] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0253] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0254] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0255] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A data query method, characterized in that, include: Receive hidden query data requests sent by the service channel client; The hidden query data request includes the user sub-identifier group generated by the service channel client; The corresponding queried data is obtained from the auxiliary database according to the hidden query data request; wherein, the queried data is the result of a near real-time backup of the data in the background database in the auxiliary database; Based on the hidden query data request, the queried data is processed using a hidden query, and the result of the hidden query is returned to the service channel client. Obtain the full data under the query channel corresponding to the user sub-identifier group as the queried data, wherein the full data includes key-value pairs of user identifier and user information; A user identifier score list is established based on the full data; Receive user sub-identifier groups sent by the service channel client; wherein, the user sub-identifier groups are generated by the service channel client by merging the identifier of the user to be queried with the identifiers of multiple randomly selected users into a user group, and then splitting each user identifier in the user group into N user sub-identifiers through key sharing; Perform a hidden query calculation on each user sub-identifier in the user sub-identifier group and each user identifier in the user identifier score list to obtain a calculation result group; The hidden query calculation is performed according to the parity classification of the second subscript of the user sub-identifier: When the second subscript of the user sub-identifier is odd, the user sub-identifier is calculated using the first type of operation rule, which is subtraction operation; When the second subscript of the user sub-identifier is even, the user sub-identifier is calculated using the second type of operation rule, which is addition. Each user sub-identifier is calculated one by one with each user identifier in the user identifier score list to obtain a calculation result group formed by the intermediate difference and the encryption result; wherein, the encryption result is the user information raised to the power of the intermediate difference. According to the order of each user sub-identifier in the user sub-identifier group, the calculation result group is returned to the service channel client; the service channel client extracts the calculation result group corresponding to the user sub-identifier based on the received calculation result group, filters the items whose difference between the sum of intermediate differences and the user identifier in the user identifier score list is 0, obtains the calculation result group corresponding to the item, and multiplies the corresponding encrypted result to obtain the query result.

2. The data query method according to claim 1, characterized in that, The step of retrieving the corresponding queried data from the auxiliary database according to the hidden query data request includes: Based on the hidden query data request, determine whether the queried data corresponding to the identifier in the user sub-identifier group belongs to hot data; If so, the corresponding hot data is read from the distributed cache of the auxiliary database and used as the queried data; wherein, the hot data is periodically stored in the distributed cache; If not, read non-hotspot data from the unbuffered storage of the auxiliary database as the queried data.

3. The data query method according to claim 2, characterized in that, Also includes: The service receives data write requests sent by the client; Write the corresponding data to the backend database according to the data write request; The data in the background database is backed up to the auxiliary database in near real-time.

4. The data query method according to claim 3, characterized in that, The near real-time backup of data from the background database to the auxiliary database includes: The host replication interface is invoked in near real-time to perform data replication operations on the host database in the background database; According to the data application requirements, the copied host database data is saved to the auxiliary database.

5. The data query method according to claim 3, characterized in that, The near real-time backup of data from the background database to the auxiliary database includes: The platform database replication interface is invoked in near real-time to perform data replication operations on the platform database in the background database; The copied platform database data is saved to the auxiliary database according to the data application requirements.

6. A data query device, characterized in that, include: The query request receiving unit is used to receive hidden query data requests sent by the service channel client; The hidden query data request includes the user sub-identifier group generated by the service channel client; The query data reading unit is used to obtain the corresponding queried data from the auxiliary database according to the hidden query data request; wherein, the queried data is the result of a near real-time backup of the data in the background database in the auxiliary database; The hidden query processing unit is used to perform hidden query processing on the queried data based on the hidden query data request, and return the hidden query processing result to the service channel client; The hidden query processing unit includes: The calculation result generation module is used to perform hidden query calculations on each identifier in the user sub-identifier group and the queried data to obtain the calculation results corresponding to each identifier, specifically including: Obtain the full data under the query channel corresponding to the user sub-identifier group as the queried data, wherein the full data includes key-value pairs of user identifier and user information; A user identifier score list is established based on the full data; Receive user sub-identifier groups sent by the service channel client; wherein, the user sub-identifier groups are generated by the service channel client by merging the identifier of the user to be queried with the identifiers of multiple randomly selected users into a user group, and then splitting each user identifier in the user group into N user sub-identifiers through key sharing; Perform a hidden query calculation on each user sub-identifier in the user sub-identifier group and each user identifier in the user identifier score list to obtain a calculation result group; The hidden query calculation is performed according to the parity classification of the second subscript of the user sub-identifier: When the second subscript of the user sub-identifier is odd, the user sub-identifier is calculated using the first type of operation rule, which is subtraction operation; When the second subscript of the user sub-identifier is even, the user sub-identifier is calculated using the second type of operation rule, which is addition. Each user sub-identifier is calculated one by one with each user identifier in the user identifier score list to obtain a calculation result group formed by the intermediate difference and the encryption result; wherein, the encryption result is the user information raised to the power of the intermediate difference. The result order return module returns the calculation result group to the service channel client according to the order of each user sub-identifier in the user sub-identifier group; the service channel client extracts the calculation result group corresponding to the user sub-identifier based on the received calculation result group, filters the items whose difference between the sum of intermediate differences and the user identifier in the user identifier score list is 0, obtains the calculation result group corresponding to the item, and multiplies the corresponding encrypted result to obtain the query result.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the data query method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the data query method according to any one of claims 1 to 5.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the data query method according to any one of claims 1 to 5.

Images