Data query method, device, equipment and storage medium

Abstract

The embodiment of the present application provides a data query method, device, equipment and storage medium, which obtains the user's data query request, which includes: index constraint parameters including a target database identifier and a field value. Since the search engine stores data information of the target database stored according to the index constraint parameters, the search engine is used to query the data according to the index constraint parameters, and the target data at the target query field value in the target database corresponding to the target database identifier is determined, and finally the target data corresponding to the data query request is output. In this technical solution, the data in the database is consistent with the data in the search engine, thereby improving the retrieval efficiency when the search function of the search engine is used to query the data.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of big data technology, and in particular to a data query method, device, equipment and storage medium. Background Art

[0002] With the rapid development of network technology, Internet applications based on the Web environment are becoming more and more widespread. In the process of enterprise informatization, various applications are built on the Web platform, which has led to a large number of development needs for Web application systems. The data of these Web application systems needs to be stored in the database on the Web platform.

[0003] In the prior art, when a database on a Web platform stores data of a Web application system, the data is generally saved through a large number of forms. Therefore, when performing subsequent data queries, a large number of query interfaces need to be called to perform precise and fuzzy queries on the data in the database forms based on various conditions.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art: when performing queries in a database form, especially fuzzy queries on multiple fields with large amounts of data, due to the large amount of data and the large number of fuzzy matching fields, there will be a problem of low data query efficiency. Summary of the invention

[0005] The embodiments of the present application provide a data query method, apparatus, device and storage medium to solve the problem of low data query efficiency in existing databases.

[0006] According to a first aspect of the present application, an embodiment of the present application provides a data query method, including:

[0007] Acquire a data query request from a user, the data query request including: an index constraint parameter including a target database identifier and a target query field value;

[0008] According to the index constraint parameters, data query is performed in a search engine to determine the target data at the target query field value in the target database corresponding to the target database identifier, wherein the search engine stores the data information of the target database stored according to the index constraint parameters;

[0009] Output the target data corresponding to the data query request.

[0010] In a possible design of the first aspect, the index constraint parameter further includes: a target query data table identifier;

[0011] The step of performing a data query in a search engine according to the index constraint parameter to determine the target data at the target query field value in the target database corresponding to the target database identifier comprises:

[0012] According to the target database identifier, locating a data table set corresponding to the target database in the search engine;

[0013] According to the target query data table identifier, locate the target query data table in the data table set corresponding to the target database;

[0014] A query is performed in the target query data table to determine the target data at the target query field value.

[0015] Optionally, the index constraint parameters also include: a target query field identifier;

[0016] The querying in the target query data table to determine the target data at the target query field value includes:

[0017] According to the target query field identifier, query the target query data table to locate the target query field;

[0018] A query is performed in the target query field to determine the target data corresponding to the target query field value.

[0019] In another possible design of the first aspect, before obtaining the user's data query request, the method further includes:

[0020] Obtaining a data processing request for the target database, the data processing request including: data to be processed and the index constraint parameter;

[0021] Based on the preconfigured database table structure and the index constraint parameters, inserting the data to be processed into the target database to generate a database log;

[0022] Transmitting the database log to the message service module through the message middleware;

[0023] Processing the database log using the message service module to obtain data update information of the target database;

[0024] Based on the data update information, the data information of the target database in the search engine is updated.

[0025] Optionally, inserting the to-be-processed data into the target database based on the preconfigured database table structure and the index constraint parameters, and generating a database log includes:

[0026] Determining a target processing data table and a target processing field in the target processing data table according to the index constraint parameters;

[0027] Based on the configuration information of the database table structure, the data to be processed is dynamically saved in the target database to generate the database log.

[0028] Optionally, the using the message service module to process the database log to obtain data update information of the target database includes:

[0029] Using the message service module, converting the database log into target processing data represented by an object;

[0030] According to the message type of the database log, packaging the target processing data into an event corresponding to the message type;

[0031] Determining, according to the type of the event, an event processing unit corresponding to the event;

[0032] The target processing data is processed using an event processing unit corresponding to the event to obtain data update information of the target database.

[0033] According to a second aspect of the present application, an embodiment of the present application provides a data query device, including:

[0034] An acquisition module is used to acquire a user's data query request, wherein the data query request includes: an index constraint parameter including a target database identifier and a target query field value;

[0035] A processing module, configured to perform data query in a search engine according to the index constraint parameters, and determine the target data at the target query field value in the target database corresponding to the target database identifier, wherein the search engine stores the data information of the target database stored according to the index constraint parameters;

[0036] The output module is used to output the target data corresponding to the data query request.

[0037] In a possible design of the second aspect, the index constraint parameter further includes: a target query data table identifier; and the processing module is specifically configured to:

[0038] According to the target database identifier, locating a data table set corresponding to the target database in the search engine;

[0039] According to the target query data table identifier, locate the target query data table in the data table set corresponding to the target database;

[0040] A query is performed in the target query data table to determine the target data at the target query field value.

[0041] Optionally, the index constraint parameters also include: a target query field identifier;

[0042] The processing module is used to query the target query data table to determine the target data at the target query field value, specifically:

[0043] The processing module is specifically used to query the target query data table according to the target query field identifier, locate the target query field, query the target query field, and determine the target data corresponding to the target query field value.

[0044] In another possible design of the second aspect, the acquisition module is further used to acquire a data processing request for the target database, the data processing request including: the data to be processed and the index constraint parameter;

[0045] The processing module is further used for:

[0046] Based on the preconfigured database table structure and the index constraint parameters, inserting the data to be processed into the target database to generate a database log;

[0047] Transmitting the database log to the message service module through the message middleware;

[0048] Processing the database log using the message service module to obtain data update information of the target database;

[0049] Based on the data update information, the data information of the target database in the search engine is updated.

[0050] Optionally, the processing module is used to insert the data to be processed into the target database based on the preconfigured database table structure and the index constraint parameters, and generate a database log, specifically:

[0051] The processing module is specifically used to determine the target processing data table and the target processing field in the target processing data table according to the index constraint parameters, dynamically save the data to be processed into the target database based on the configuration information of the database table structure, and generate the database log.

[0052] Optionally, the processing module is used to process the database log using the message service module to obtain the data update information of the target database, specifically:

[0053] The processing module is specifically used for:

[0054] Using the message service module, converting the database log into target processing data represented by an object;

[0055] According to the message type of the database log, packaging the target processing data into an event corresponding to the message type;

[0056] Determining, according to the type of the event, an event processing unit corresponding to the event;

[0057] The target processing data is processed using an event processing unit corresponding to the event to obtain data update information of the target database.

[0058] According to the third aspect of the present application, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in the first aspect and various possible designs is implemented.

[0059] According to the fourth aspect of the present application, an embodiment of the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the methods described in the above-mentioned first aspect and various possible designs.

[0060] According to the fifth aspect of the present application, an embodiment of the present application provides a computer program product, including: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the above first aspect and each possible design.

[0061] The data query method, device, equipment and storage medium provided by the embodiment of the present application obtain the data query request of the user, which includes: index constraint parameters including the target database identifier and field value. Since the search engine stores the data information of the target database stored according to the index constraint parameters, the search engine is used to query the data according to the index constraint parameters, and the target data at the target query field value in the target database corresponding to the target database identifier is determined, and finally the target data corresponding to the data query request is output. In this technical solution, the data in the database is consistent with the data in the search engine, thereby improving the retrieval efficiency when the search function of the search engine is used to query the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0063] Figure 1A A schematic diagram of an application scenario of the data query method provided in an embodiment of the present application;

[0064] Figure 1B A schematic diagram of another application scenario of the data query method provided in an embodiment of the present application;

[0065] Figure 2 A flowchart of a first embodiment of a data query method provided in an embodiment of the present application;

[0066] Figure 3 A flowchart of a second embodiment of a data query method provided in an embodiment of the present application;

[0067] Figure 4 A flowchart of a third embodiment of the data query method provided in the embodiment of the present application;

[0068] Figure 5 This is a schematic diagram of the database table structure in the embodiment of the present application;

[0069] Figure 6 A flowchart of a fourth embodiment of a data query method provided in an embodiment of the present application;

[0070] Figure 7 This is a schematic diagram of the structure of the message service module in this embodiment;

[0071] Figure 8 A schematic diagram of the structure of an embodiment of a data query device provided in an embodiment of the present application;

[0072] Fig. 9 A schematic diagram of the structure of an electronic device embodiment provided in an embodiment of the present application.

[0073] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0074] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0075] First, the terms involved in the embodiments of the present application are explained:

[0076] Database: It is a computer software system that organizes, stores and manages data according to data structure. It is a collection of large amounts of data that is stored in a computer for a long time, organized, shareable and uniformly managed. The types of databases include: relational databases and non-relational databases. Among them, the storage format of relational databases can intuitively reflect the relationship between entities. Common relational databases include Mysql, SqlServer, etc. Non-relational databases refer to distributed, non-relational data storage systems that do not guarantee compliance with the ACID principle. Among them, ACID refers to the abbreviation of the four basic elements for the correct execution of database transactions, including: atomicity, consistency, isolation, and durability.

[0077] In the embodiments of the present application, the database includes but is not limited to: sqlserver, mysql, oracle, redis, hbase and other storage systems.

[0078] Search engine: It is a search technology that uses specific strategies to retrieve specified information from the Internet and feed it back to users based on user needs and certain algorithms. Among them, the search method is a key link of the search engine, which can be roughly divided into four types: full-text search engine, meta search engine, vertical search engine and directory search engine, each of which has its own characteristics and is suitable for different search environments.

[0079] Among them, the full-text search engine is a search method that uses crawler programs to crawl all relevant articles on the Internet and index them. Its working principle is that the computer indexing program scans each word in the article and creates an index for each word, indicating the number and position of the word in the article. When the user queries, the search program searches according to the pre-established index and feeds the search results back to the user. The full-text search engine is suitable for general Internet users. This search method is convenient and simple, and it is easy to obtain all relevant information. However, the searched information is too complex, so users need to browse and identify the required information one by one.

[0080] Current mainstream full-text search engines include but are not limited to: elasticsearch, solr, and lucene.

[0081] Message queue middleware: It is an important component in large systems and has gradually become the core means of internal communication in enterprise systems. It has a series of functions such as loose coupling, asynchronous messaging, traffic peak shaving, reliable delivery, broadcasting, flow control, and eventual consistency. It has become one of the main means of asynchronous remote procedure call (RPC). At present, common message queue middlewares include ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, RocketMQ, etc.

[0082] It can be seen from the content of the above background technology that when querying in a database form, especially fuzzy query of multiple fields with large data volume, there will be a problem of low data query efficiency due to the large amount of data and the large number of fuzzy matching fields. To address this problem, the inventors have found that if operations such as adding, modifying, and deleting data in the database are mapped to a search engine, and the search engine's retrieval is used instead of the database query, the problem of slow response of multi-field fuzzy query under large data volume can be solved, thereby improving the response speed of web applications.

[0083] Furthermore, by configuring the database table structure, http interfaces for adding, modifying, deleting, and querying can be provided to improve development efficiency and R&D quality.

[0084] Specifically, the technical solution of the present application is: when the search engine stores data information of the target database stored according to the index constraint parameters, when the electronic device obtains the user's data query request, it can perform data query in the search engine according to the index constraint parameters containing the target database identifier in the data query request, and can determine the target data corresponding to the data query request. That is, when the data information of the target database is mapped to the search engine, it can ensure that the database and the search engine data are consistent, so that when the search engine's retrieval function is used to replace the database's fuzzy query, it can ensure that excellent multi-field fuzzy query performance can still be provided under extremely large data volumes (tens of billions), thereby improving retrieval efficiency.

[0085] Before introducing the technical solution of the present application, the application scenario of the technical solution of the present application is first introduced. Figure 1A A schematic diagram of an application scenario of the data query method provided in the embodiment of the present application. Figure 1AAs shown, the scenario diagram may include: a user terminal 11 and an electronic device 12. The electronic device 12 may include: a configuration module 121, a database 122, a message middleware 123, a message service module 124 and a search engine 125.

[0086] Optionally, when a user has a data processing request for a database, the user terminal 11 can be used to submit a data processing request to the electronic device 12. The configuration module 121 of the electronic device 12 can obtain the data processing request and dynamically update the data information stored in the database 122 according to the preset database structure, the index constraint parameters carried in the data processing request and the data to be processed, thereby generating a database log, and transmitting the database log to the message service module 124 through the message middleware 123. The message service module 124 processes the received database log and updates the data update information of the database 122 to the search engine 125.

[0087] Correspondingly, when a user has a data query request, the user can send the data query request to the electronic device 12 through the user terminal 11, so that the search engine 125 of the electronic device 12 can directly obtain the data query request, and based on the index constraint parameters of the data to be queried carried in the data query request, perform data query in the search engine, and can directly determine the target data corresponding to the data query request and execute the data export process.

[0088] Figure 1B A schematic diagram of another application scenario of the data query method provided in an embodiment of the present application. Figure 1B and Figure 1A The difference is that the user can directly operate the electronic device 12, for example, the electronic device 12 has a user operation interface 120. Therefore, when the user has a data processing request or a data query request for the database, the user can directly operate the user operation interface 120, so that the electronic device 12 obtains the data processing request or the data query request through the user operation interface 120.

[0089] Understandably, Figure 1B The electronic device 12 and Figure 1A The components of the electronic device may be the same. Figure 1B In the application scenario shown, the electronic device 12 may also include: a configuration module 121, a database 122, a message middleware 123, a message service module 124 and a search engine 125. For the functions of each component, see Figure 1A The records in the application scenario shown will not be repeated here.

[0090] It should be noted that Figure 1A and Figure 1BThese are just two exemplary application scenario diagrams provided in the embodiments of the present application. The devices included in each application scenario can be set according to actual needs and will not be described in detail here.

[0091] Optionally, the data query method provided in the embodiment of the present application is Figure 1A or Figure 1B The electronic device in the scene diagram is explained as an execution subject. Generally, the electronic device is a server, or a data storage device dedicated to storing data, such as a terminal device, which is not limited in the embodiments of the present application.

[0092] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0093] Figure 2 The flowchart of the first embodiment of the data query method provided in the embodiment of the present application is as follows. Figure 2 As shown, the data query method may include the following steps:

[0094] S201. Obtain a data query request from a user, where the data query request includes index constraint parameters including a target database identifier and a field value.

[0095] As an example, when a user has a data query requirement, the user can directly operate the electronic device to issue a data query request. For example, when the electronic device has a user operation interface, the user can issue a data query request by operating the user operation interface; when the electronic device has a voice receiving function, the data query request can also be issued by voice, so that the electronic device can obtain the data query request by using the voice receiving function.

[0096] As another example, when the electronic device is connected to the user terminal, the user may send a data query request to the electronic device through the user terminal, so that the electronic device obtains the data query request.

[0097] This embodiment does not limit the manner in which the electronic device obtains the data query request, which can be determined according to actual needs and will not be elaborated here.

[0098] In actual applications, since the index constraint parameters are stored in the format of database identifier, data table identifier (also called data table alias), field identifier (also called field alias) (optional) and field value, different databases, different fields and different field values ​​can be located based on different index constraint parameters, and then the data information in different databases can be determined. Therefore, in this embodiment, in order to enable the user to accurately locate the target data to be queried, the data query request issued by the user needs to carry index constraint information, and the index constraint information can include the target database identifier and the target query field value.

[0099] S202. According to the above index constraint parameters, data query is performed in a search engine to determine the target data at the target query field value in the target database corresponding to the target database identifier. The search engine stores data information of the target database stored according to the index constraint parameters.

[0100] Exemplarily, the search engine of the electronic device stores data information of different databases stored according to different index constraint parameters. The data in the search engine can be stored in an inverted index format. Therefore, when the electronic device performs data retrieval, the index constraint parameters are first parsed to determine the target database identifier and target query field value that the index constraint parameters at least include. Then, a full-text search is performed in the search engine based on the target database identifier and the target query field value to obtain the target data corresponding to the data query request.

[0101] For the specific implementation of this step, please refer to the following Figure 3 The description in the illustrated embodiment will not be repeated here.

[0102] S203: Output the target data corresponding to the data query request.

[0103] As an example, when the electronic device has a user operation interface, the target data corresponding to the data query request can be displayed on the user operation interface to be presented to the user.

[0104] As another example, the electronic device may also send the target data corresponding to the data query request to the user terminal, so that the user terminal presents the determined target data to the user.

[0105] The data query method provided by the embodiment of the present application obtains the data query request of the user, and the data query request includes: index constraint parameters including the target database identifier and field value. Since the search engine stores the data information of the target database stored according to the index constraint parameters, the search engine is used to perform data query according to the index constraint parameters, and the target data at the target query field value in the target database corresponding to the target database identifier is determined, and finally the target data corresponding to the data query request is output. In this technical solution, the data in the database is consistent with the data in the search engine, thereby improving the retrieval efficiency when the search engine's retrieval function is used to perform data query.

[0106] For example, in a possible design of the embodiment of the present application, the above index constraint parameters also include: target query data table identifier, target query field value. At this time, the above S202 can be performed as follows Figure 3 The embodiment shown implements.

[0107] Figure 3 The flowchart of the second embodiment of the data query method provided in the embodiment of the present application is as follows. Figure 3 As shown, the above S202 can be implemented by the following steps:

[0108] S301. According to the target database identifier, locate the data table set corresponding to the target database in the search engine.

[0109] In this embodiment, the search engine of the electronic device stores data from multiple databases, each database has an identifier, and the data of each database is stored in the search engine according to index constraint parameters in the format of database identifier, data table identifier (also called data table alias), field identifier (also called field alias) (optional) and field value. Therefore, when the electronic device obtains a data query request, it first parses the target database identifier from the data query request, and then locates the data table set corresponding to the target database in the search engine based on the target database identifier.

[0110] S302: According to the target query data table identifier, locate the target query data table in the data table set corresponding to the target database.

[0111] Optionally, the data of each database may be stored in multiple data tables. Therefore, after determining the target query data table identifier from the index constraint parameters, a search query can be performed in the data table set corresponding to the target database to locate the target query data table to be queried.

[0112] In an example of the present application, the electronic device can directly query the target query data table to determine the target data at the target query field value. For example, when the index constraint parameter does not include the target query field identifier, the electronic device can directly query the target query data table based on the target query field value, directly locate the target query field value, and thus obtain the target data at the target query field value.

[0113] In another example of the present application, when the index constraint parameter further includes: when the target query field identifier, querying in the target query data table to determine the target data at the target query field value can be implemented by the following steps;

[0114] S303: According to the target query field identifier, query is performed in the target query data table to locate the target query field.

[0115] In this step, when the index constraint parameter includes a target query field identifier, the electronic device can locate the target query field in the target query data table according to the target query field identifier, thereby laying a foundation for acquiring the target data.

[0116] S304: Query the target query field to determine the target data corresponding to the target query field value.

[0117] Exemplarily, data in the search engine is stored according to index constraint parameters (key), and different keys correspond to different data index information. Therefore, when locating the target query field, the electronic device can locate the location of the target query field value, thereby obtaining the target data corresponding to the target query field value.

[0118] In the data query method provided by the embodiment of the present application, if the index constraint parameters include a target database identifier, a target query data table identifier, and a target field value, then according to the target database identifier, in the search engine, the data table set corresponding to the target database can be located, and then according to the target query data table identifier, in the data table set corresponding to the target database, the target query data table can be located, and the target query data table can be queried in the target query data table to determine the target data at the target query field value; further, if the index constraint parameters also include a target query field identifier, firstly according to the target query field identifier, the target query field can be located in the target query data table, and then the target query field can be queried in the target query field to determine the target data corresponding to the target query field value. This technical solution can query the target data in the search engine based on the index constraint parameters, avoids the fuzzy query method in the database, and improves the data query efficiency.

[0119] Exemplarily, in a possible design of an embodiment of the present application, the electronic device is able to perform data query in a search engine based on the index constraint parameters carried in the data query request on the premise that relevant data information is stored in the search engine. Therefore, before the above S201, the technical solution also needs to include a data processing process, such as data storage, update, deletion and other operations.

[0120] Optional, Figure 4 The flowchart of the third embodiment of the data query method provided in the embodiment of the present application is as follows. Figure 4 As shown, before the above S201, the method may further include the following steps:

[0121] S401: Obtain a data processing request for a target database, where the data processing request includes: data to be processed and index constraint parameters.

[0122] As an example, when a user has a data operation requirement (e.g., data storage, update, deletion, etc.) for a target database, a data processing request can be issued by operating an electronic device. As another example, when a user operates a web application on a terminal device, certain data is also generated, and a data processing request is generated accordingly, so as to update the database corresponding to the web application based on the data.

[0123] It is understandable that in order for the electronic device to determine the object to be processed, the data processing request will include the data to be processed. In order to ensure the processing location of the data to be processed, the data processing request will also include an index constraint parameter, which carries the identifier, field identifier and field value of the data table.

[0124] S402: Based on the preconfigured database table structure and the above index constraint parameters, insert the data to be processed into the target database and generate a database log.

[0125] In an embodiment of the present application, a configuration module is provided in the electronic device, and the core function of the configuration module is to store the database table structure so that the electronic device can process the database data, such as query, export, insert, modify, delete and other operations.

[0126] For example, when a user needs to store data in a database, the electronic device can dynamically save the data to be processed into the database according to a preconfigured database table structure when acquiring the data to be processed submitted by the user, thereby generating a database log.

[0127] Exemplarily, S402 may be implemented by the following steps:

[0128] A1. Determine the target processing data table and the target processing field in the target processing data table according to the index constraint parameters;

[0129] A2. Based on the configuration information of the database table structure, the data to be processed is dynamically saved in the target database to generate the database log.

[0130] Optionally, after the configuration module of the electronic device obtains the data processing request, it extracts the index constraint parameters from the data processing request, and then extracts the data table identifier and the field identifier from the index constraint parameters, so as to find the target processing data table and locate the target processing field corresponding to the data table. Finally, based on the database table structure, the data to be processed in the data processing request is stored in the target processing data table and the target processing field in the target processing data table, thereby realizing data storage. Correspondingly, the target database will generate a database log.

[0131] For example, Figure 5 Schematic diagram of the database table structure in the embodiment of the present application. Figure 5 As shown, the configuration module is centered on the data table in the database, and is configured with information such as the field information, query conditions, query structure, and export tasks of the data table.

[0132] The data table refers to the name of the data table in the record database used to configure the interface and has a specified alias (i.e., an identifier);

[0133] Field refers to the field name, field alias, field type, etc. corresponding to the data table in the record database;

[0134] Export tasks refer to tasks that record user exported data;

[0135] Query conditions refer to the condition names required for storage export tasks and query tasks, including field identifiers, aliases of query condition fields, etc.

[0136] Query results refer to the field identifiers and field aliases that need to be returned by storage export tasks and query tasks.

[0137] In this embodiment, by configuring Figure 5 The data table shown can dynamically process the data in the data table, and can also quickly provide http interfaces for adding, modifying, deleting, and querying through some simple configurations. This can better solve the interaction needs between different systems (which may have different functions, different development languages, different service providers, etc., but are all systems based on http services), thus laying the foundation for subsequent data queries.

[0138] S403: Transmit the database log to the message service module through the message middleware.

[0139] In this embodiment, when the data in the database changes, a database log will be generated and output. In order to make the data in the target database stored by the search engine consistent with the data in the target database, the electronic device can extract the database log to the message middleware after generating the database log, and then transmit it to the message service module through the message middleware, so that the message service module can process the database log and save it to the search engine.

[0140] Optionally, message middleware is also called message queue middleware, which is used for data communication between applications and has a series of functions such as loose coupling, asynchronous messaging, traffic peak shaving, reliable delivery, broadcasting, flow control, and eventual consistency. It is the core means of internal communication in an enterprise. Therefore, in this embodiment, using message middleware to transmit database logs can ensure that the message service module can smoothly perform data analysis and processing.

[0141] S404: Use the message service module to process the database log to obtain data update information of the target database.

[0142] Optionally, JavaScript Object Notation (JSON) has a concise and clear hierarchical structure, making JSON an ideal data exchange language. JSON is a lightweight data exchange format that uses a text format that is completely independent of programming languages ​​to store and represent data. It is easy for people to read and write, and is also easy for machines to parse and generate, and effectively improves network transmission efficiency. Therefore, after obtaining the database log, the message service module can parse the database log into JSON data and save the JSON data to the search engine.

[0143] It can be understood that, in this embodiment, the JSON data generated by the message service module after processing the database log is the data update information of the target database, which will not be described in detail here.

[0144] S405: Based on the above data update information, update the data information of the target database in the search engine.

[0145] In this step, after obtaining the data update information of the target database, the electronic device may update the data update information to the search engine so that the data in the search engine is consistent with the data in the target database.

[0146] For example, the message service module receives a database log from the message middleware (the database log can be a data insertion log, a data update log, or a data deletion log, which is not limited in this embodiment), restores the database log into real updated data information, and writes it into the search engine so that the search engine can provide accurate retrieval services.

[0147] The data query method provided by the embodiment of the present application obtains a data processing request for a target database, inserts the data to be processed into the target database based on the preconfigured database table structure and index constraint parameters, generates a database log, transmits the database log to the message service module through the message middleware, and then uses the message service module to process the database log to obtain the data update information of the target database, and finally updates the data information of the target database in the search engine based on the data update information. In this technical solution, by mapping the database log generated by the database to the search engine, it can ensure that the data of the database and the search engine are consistent, so that when the search engine's retrieval function is used to replace the fuzzy query of the database in the future, it can ensure that excellent data query performance can still be provided under extremely large data volumes (tens of billions).

[0148] Furthermore, in an embodiment of the present application, the message service module is mainly used to receive database logs from the message middleware and restore the database logs into real data, thereby obtaining data update information of the target database and writing it to the search engine to provide retrieval services.

[0149] For example, in the above Figure 4 Based on the embodiment shown, Figure 6 This is a flow chart of the fourth embodiment of the data query method provided in the present application. Figure 6 As shown, the above S404 can be implemented by the following steps:

[0150] S601. Using a message service module, convert database logs into target processing data represented by objects.

[0151] In the embodiment of the present application, since the text format of the object representation can store and represent data completely independently of the programming language, when the electronic device receives the database log using the message service module, it can convert it into the target processing data represented by the object, for example, data in JSON format, and then perform different processing operations based on the message type of the database log.

[0152] S602: According to the message type of the database log, the target processing data is packaged into an event corresponding to the message type.

[0153] Optionally, in this embodiment, the message service module can analyze the database log, determine the message type of the database log, and then package the target processing data converted in the above S601 into different events based on the message type of the database log, so that it can be transferred to different processing units for processing according to the event type.

[0154] S603: Determine an event processing unit corresponding to the event according to the type of the event.

[0155] Optionally, in an embodiment of the present application, the message service module may include an event processing unit for processing different event types, so that the electronic device can determine the event processing unit for processing the event according to the type of the above event, for example, an insert event, an update event, a delete event, etc., such as an insert event processing unit, an update event processing unit, and a delete event processing unit.

[0156] S604: Utilize the event processing unit corresponding to the event to process the target processing data to obtain data update information of the target database.

[0157] Optionally, after determining the event processing unit corresponding to the event, the message service module can start the corresponding event processing unit to perform data processing. For example, the insert event processing unit can be used to process the insert event, the update event processing unit can be used to process the update event, the delete event processing unit can be used to process the delete event, etc. When the event processing unit completes processing the target data, the data update information of the target database can be obtained.

[0158] For example, Figure 7 FIG. 1 is a schematic diagram of the structure of the message service module in this embodiment. Figure 7 As shown, the message service module may include a message receiving unit, an event processing unit routing, an event encapsulation unit, an event registration unit, an event processing unit, etc. Among them, the event processing unit can be divided into different types of event processing units according to the event type. For example, when the event type includes an insert event, an update event, and a delete event, the event processing unit may include an insert event processing unit, an update event processing unit, and a delete event unit, etc.

[0159] exist Figure 7 In the structural diagram shown, when the electronic device works normally, the message service module is started, and all event processing units included in the message service module are registered in the event registration unit, so that the event registration unit manages all event processing units.

[0160] Optionally, the message receiving unit of the message service module is connected to the above-mentioned message middleware, and is able to receive database logs from the message middleware, and transmit them to the event processing unit routing for data conversion to obtain target processing data represented by the object. The event processing unit routing transmits the target processing data to the event encapsulation unit, and the event encapsulation unit packages the target processing data into an event corresponding to the message type, and determines the task identifier corresponding to the event, such as an insert event, an update event or a delete event, etc., and then according to the task identifier, transmits the event to the event processing unit through the event processing unit routing, and assigns it to the event processing unit corresponding to the task identifier for processing to obtain data update information of the target database.

[0161] The data query method provided by the embodiment of the present application uses a message service module to convert a database log into target processing data represented by an object, and according to the message type of the database log, the target processing data is packaged into an event corresponding to the message type, and according to the type of the event, the event processing unit corresponding to the event is determined, and finally the event processing unit corresponding to the event is used to process the target processing data to obtain data update information of the target database. In this technical solution, the message service module can ensure the accuracy and efficiency of data processing by distinguishing between events and event processing units.

[0162] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0163] Figure 8 This is a schematic diagram of the structure of the data query device embodiment provided in the present application. Figure 8 Said data query device may include:

[0164] The acquisition module 801 is used to acquire a user's data query request, wherein the data query request includes: an index constraint parameter including a target database identifier and a target query field value;

[0165] Processing module 802, for performing data query in a search engine according to the index constraint parameters, and determining the target data at the target query field value in the target database corresponding to the target database identifier, wherein the search engine stores the data information of the target database stored according to the index constraint parameters;

[0166] The output module 803 is used to output the target data corresponding to the data query request.

[0167] In a possible design of the embodiment of the present application, the index constraint parameter further includes: a target query data table identifier;

[0168] The processing module 802 is specifically configured to:

[0169] According to the target database identifier, locating a data table set corresponding to the target database in the search engine;

[0170] According to the target query data table identifier, locate the target query data table in the data table set corresponding to the target database;

[0171] A query is performed in the target query data table to determine the target data at the target query field value.

[0172] Optionally, the index constraint parameters also include: a target query field identifier;

[0173] Processing module 802 is used to query the target query data table to determine the target data at the target query field value, specifically:

[0174] The processing module 802 is specifically configured to query the target query data table according to the target query field identifier, locate the target query field, query the target query field, and determine the target data corresponding to the target query field value.

[0175] In another possible design of the embodiment of the present application, the acquisition module 801 is further used to acquire a data processing request for the target database, the data processing request including: the data to be processed and the index constraint parameter;

[0176] The processing module 802 is further configured to:

[0177] Based on the preconfigured database table structure and the index constraint parameters, inserting the data to be processed into the target database to generate a database log;

[0178] Transmitting the database log to the message service module through the message middleware;

[0179] Processing the database log using the message service module to obtain data update information of the target database;

[0180] Based on the data update information, the data information of the target database in the search engine is updated.

[0181] Optionally, the processing module 802 is used to insert the data to be processed into the target database based on the preconfigured database table structure and the index constraint parameters, and generate a database log, specifically:

[0182] The processing module 802 is specifically used to determine the target processing data table and the target processing field in the target processing data table according to the index constraint parameters, dynamically save the data to be processed into the target database based on the configuration information of the database table structure, and generate the database log.

[0183] Optionally, the processing module 802 is used to process the database log using the message service module to obtain data update information of the target database, specifically:

[0184] The processing module 802 is specifically used for:

[0185] Using the message service module, converting the database log into target processing data represented by an object;

[0186] According to the message type of the database log, packaging the target processing data into an event corresponding to the message type;

[0187] Determining, according to the type of the event, an event processing unit corresponding to the event;

[0188] The target processing data is processed using an event processing unit corresponding to the event to obtain data update information of the target database.

[0189] The device provided in the embodiment of the present application can be used to execute the technical solution described in the above method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0190] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the processing module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0191] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0192] Fig. 9 This is a schematic diagram of the structure of an electronic device embodiment provided in the present application. Fig. 9 As shown, the electronic device may include: a processor 901, a memory 902, a communication interface 903 and a system bus 904, wherein the memory 902 and the communication interface 903 are connected to the processor 901 through the system bus 904 and communicate with each other, the memory 902 is used to store computer instructions, the communication interface 903 is used to communicate with other devices, and the processor 901 implements the technical solution described in the above method embodiment when executing the computer instructions.

[0193] Optionally, in an embodiment of the present application, the electronic device may further include a user operation interface 905, and the user operation interface 905 may be used to receive a user's data query request or other data processing request.

[0194] In this Fig. 9 In the embodiment, the processor 901 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components.

[0195] The memory 902 may include a random access memory (RAM), may include a read-only memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0196] The communication interface 903 is used to implement the communication between the database access device and other devices (such as the client, the read-write library and the read-only library).

[0197] The system bus 904 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0198] According to an embodiment of the present application, the embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the technical solution described in the above method embodiment.

[0199] According to an embodiment of the present application, the embodiment of the present application also provides a chip for running instructions, and the chip is used to execute the technical solution described in the above method embodiment.

[0200] According to an embodiment of the present application, the present application also provides a computer program product, including: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0201] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the application disclosed herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0202] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data query method, characterized in that: include: Acquire a data query request from a user, the data query request including: an index constraint parameter including a target database identifier and a target query field value; According to the index constraint parameters, data query is performed in a search engine to determine the target data at the target query field value in the target database corresponding to the target database identifier, wherein the search engine stores the data information of the target database stored according to the index constraint parameters; Outputting target data corresponding to the data query request; Before obtaining the user's data query request, the method further includes: Obtaining a data processing request for the target database, the data processing request including: data to be processed and the index constraint parameter; Based on the preconfigured database table structure and the index constraint parameters, inserting the data to be processed into the target database to generate a database log; Transmitting the database log to the message service module through the message middleware; Processing the database log using the message service module to obtain data update information of the target database; Based on the data update information, the data information of the target database in the search engine is updated so that the data information in the search engine is consistent with the data in the target database.

2. The method according to claim 1, characterized in that The index constraint parameters also include: a target query data table identifier; The step of performing a data query in a search engine according to the index constraint parameter to determine the target data at the target query field value in the target database corresponding to the target database identifier comprises: According to the target database identifier, locating a data table set corresponding to the target database in the search engine; According to the target query data table identifier, locate the target query data table in the data table set corresponding to the target database; A query is performed in the target query data table to determine the target data at the target query field value.

3. The method according to claim 2, characterized in that The index constraint parameters also include: a target query field identifier; The querying in the target query data table to determine the target data at the target query field value includes: According to the target query field identifier, query the target query data table to locate the target query field; A query is performed in the target query field to determine the target data corresponding to the target query field value.

4. The method according to claim 1, characterized in that The inserting the to-be-processed data into the target database based on the pre-configured database table structure and the index constraint parameters, and generating a database log, comprises: Determining a target processing data table and a target processing field in the target processing data table according to the index constraint parameters; Based on the configuration information of the database table structure, the data to be processed is dynamically saved in the target database to generate the database log.

5. The method according to claim 1, characterized in that The using the message service module to process the database log to obtain data update information of the target database includes: Using the message service module, converting the database log into target processing data represented by an object; According to the message type of the database log, packaging the target processing data into an event corresponding to the message type; Determining, according to the type of the event, an event processing unit corresponding to the event; The target processing data is processed using an event processing unit corresponding to the event to obtain data update information of the target database.

6. A data query device, characterized in that: include: An acquisition module is used to acquire a user's data query request, wherein the data query request includes: an index constraint parameter including a target database identifier and a target query field value; A processing module, configured to perform data query in a search engine according to the index constraint parameters, and determine the target data at the target query field value in the target database corresponding to the target database identifier, wherein the search engine stores the data information of the target database stored according to the index constraint parameters; An output module, used for outputting target data corresponding to the data query request; The acquisition module is further used to acquire a data processing request for the target database, wherein the data processing request includes: data to be processed and the index constraint parameter; The processing module is further used for: Based on the preconfigured database table structure and the index constraint parameters, inserting the data to be processed into the target database to generate a database log; Transmitting the database log to the message service module through the message middleware; Processing the database log using the message service module to obtain data update information of the target database; Based on the data update information, the data information of the target database in the search engine is updated so that the data information in the search engine is consistent with the data in the target database.

7. The device according to claim 6, characterized in that The index constraint parameters also include: a target query data table identifier; the processing module is specifically used to: According to the target database identifier, locating a data table set corresponding to the target database in the search engine; According to the target query data table identifier, locate the target query data table in the data table set corresponding to the target database; A query is performed in the target query data table to determine the target data at the target query field value.

8. The device according to claim 7, characterized in that The index constraint parameters also include: a target query field identifier; The processing module is used to query the target query data table to determine the target data at the target query field value, specifically: The processing module is specifically used to query the target query data table according to the target query field identifier, locate the target query field, query the target query field, and determine the target data corresponding to the target query field value.

9. The device according to claim 6, characterized in that The processing module is used to insert the data to be processed into the target database based on the preconfigured database table structure and the index constraint parameters, and generate a database log, specifically: The processing module is specifically used to determine the target processing data table and the target processing field in the target processing data table according to the index constraint parameters, dynamically save the data to be processed into the target database based on the configuration information of the database table structure, and generate the database log.

10. The device according to claim 6, characterized in that The processing module is used to process the database log using the message service module to obtain the data update information of the target database, specifically: The processing module is specifically used for: Using the message service module, converting the database log into target processing data represented by an object; According to the message type of the database log, packaging the target processing data into an event corresponding to the message type; Determining, according to the type of the event, an event processing unit corresponding to the event; The target processing data is processed using an event processing unit corresponding to the event to obtain data update information of the target database.

11. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 5 when executed by a processor.

13. A computer program product comprising: A computer program, characterized in that when the computer program is executed by a processor, it is used to implement the method according to any one of claims 1 to 5.

Images