Complex retrieval method in big data scene based on transactional database

Abstract

The invention discloses a complex retrieval method in a big data scene based on a transactional database, and belongs to the technical field of data storage management. The implementation method comprises the following steps that: 1, a transactional database receives a transactional request and performs a transactional operation; 2, the middleware service monitors the database subjected to transaction operation processing; 3, forming an event message for full-text retrieval engine consumption; 4, the full-text retrieval engine synchronizes data operation to a search engine by using the details of the data operation, and constructs an index structure of inverted sorting of hot data; 5, setting a threshold value, compressing and archiving the cold data in a timing manner, and storing the cold data in an object storage service; forming hot data from the data with the access frequency and effectiveness higher than thresholds; and 6, when the application service performs data query, obtaining a query result according to the query complexity. Compared with the prior art, the technical problem of high time delay caused by complex retrieval of a transactional database in a big data scene is solved.

Description

Technical Field

[0001] The present invention relates to a complex retrieval method based on a transactional database big data scenario, and belongs to the technical field of data storage management. Background Art

[0002] At present, the current status of data retrieval based on transactional databases in big data scenarios is as follows: transactional databases were originally designed for operations on highly consistent, highly isolated, and relational data, but big data scenarios focus more on high throughput, high concurrency, and rapid analysis of large-scale data. Therefore, in big data scenarios, data retrieval based on transactional databases faces multiple challenges. When the amount of data reaches billions of rows or TB levels, the execution time of a single query increases dramatically, and the query time is extended from milliseconds to several minutes, or even timeouts. In a high-concurrency environment, the waiting time of transactions increases, SQL execution is stuck, and even deadlocks occur.

[0003] In the tide of globalization, traditional transactional databases can no longer meet the big data usage scenarios of complex modern applications. When facing big data scenarios, relational databases can improve performance, scalability, and high availability by sharding or directly using high-availability distributed databases. However, when facing query tasks in complex scenarios, these databases still cannot meet the application's demand for fast and efficient databases.

[0004] Therefore, the high latency caused by complex retrieval of transactional databases in big data scenarios has become a problem that needs to be solved urgently. Summary of the invention

[0005] In view of the high latency problem caused by complex retrieval of transactional databases in big data scenarios, the purpose of the present invention is to provide a complex retrieval method based on transactional databases in big data scenarios to improve the efficiency of complex retrieval in big data scenarios.

[0006] The working principle of the present invention is as follows: transactional databases include relational data, document databases, and any database that supports transactional operations; data changes can be obtained by monitoring database logs or by actively sending change notifications through application services; after the middleware obtains data changes and performs cleaning and sorting, it can send messages to a message queue to relieve the service pressure of the full-text search engine and increase service reliability; the complex retrieval method, storage, and system of the transactional database of the present invention in a big data scenario realize the strong consistency and reliability requirements of the transactional database in a big data scenario, solve the complex retrieval problem in a big data scenario, and avoid query failure, timeout, and other problems under complex retrieval conditions such as fuzzy retrieval, multi-value retrieval, and exclusive retrieval, as well as cross-regional slow query problems.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] The present invention discloses a complex retrieval method based on a transactional database big data scenario, comprising the following steps:

[0009] Step 1: The transactional database receives transactional requests for database data from application services and performs transaction operations;

[0010] Step 2: The middleware service monitors the database after the transaction operation is executed to further obtain the data operation details; and uses the data operation details to clean and organize the data;

[0011] Step 3: The middleware service encapsulates the cleaned and sorted data to form event messages for consumption by the full-text search engine;

[0012] Step 4: The full-text search engine uses the data operation details to synchronize the data operation to the search engine and build an index structure with hot data sorted in reverse order;

[0013] Step 5: Set a threshold, and convert data with access frequency and effectiveness below the threshold into cold data; compress and archive the cold data in a scheduled manner and store it in the object storage service; convert data with access frequency and effectiveness above the threshold into hot data;

[0014] Step 6: When the application service performs data query, the query results are obtained according to the query complexity;

[0015] Step 6.1: When the application service performs a hot data query, the query results are obtained through the transactional database and full-text search engine;

[0016] Step 6.2: When the application service performs cold data query, it obtains archived data from the object storage service, decompresses it, creates an index, and performs query;

[0017] The present invention discloses a complex retrieval system based on a transactional database big data scenario, which is used to implement the above method. The present invention discloses a complex retrieval system based on a transactional database big data scenario, which includes a transactional database unit, a data change processing unit, and a data indexing unit;

[0018] The transactional database unit is used to store application data, provide atomicity, consistency, isolation and persistence for data operations, and ensure strong consistency and reliability requirements for data operations; it will serve as the input of the data change processing unit;

[0019] The data change processing unit is used to monitor changes in the database, including addition, update, and deletion operations, and to clean and organize the data;

[0020] The data indexing unit, as the output of the data change processing unit, performs word segmentation on the data content, shields invalid characters, and then constructs an inverted index for subsequent complex retrieval services;

[0021] A computer-readable storage medium, the computer-readable storage medium is used to store a program of a data transmission method based on a cloud object storage service, the program of the data transmission method based on the cloud object storage service is used to be executed; the instructions of the data transmission method based on the user datagram protocol are used to implement the complex retrieval method based on a transactional database big data scenario;

[0022] An electronic device, comprising a memory and a processor, wherein the processor executes a program of the data transmission method based on a cloud object storage service to implement the complex retrieval method based on a transactional database big data scenario;

[0023] Beneficial effects:

[0024] Compared with the prior art, it has the following beneficial effects:

[0025] 1. The transactional database of the present invention needs to support index creation in order to speed up simple queries.

[0026] 2. The middleware of the present invention uses database log monitoring to obtain data changes in the database so as to eliminate the impact on the effectiveness of application operations.

[0027] 3. The message of the present invention is sent to the message queue through the middleware in the form of binary byte stream to accelerate the encoding and decoding speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A complex retrieval method, storage and system flow chart based on a transactional database in a big data scenario provided in Example 1 of the present invention;

[0029] Figure 2 This is a diagram of a complex retrieval method, storage and system architecture based on a transactional database in a big data scenario provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0030] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the protection scope of the present invention.

[0031] The English abbreviations in the following embodiments have the following meanings:

[0032] MySQL: relational database management system.

[0033] Canal: database incremental subscription and consumption component;

[0034] Kafka - open source distributed event streaming platform.

[0035] Elasticsearch - full-text search engine.

[0036] Example

[0037] like Figure 1 As shown, the present embodiment is a complex retrieval method based on a transactional database big data scenario, and the specific implementation steps are as follows:

[0038] Step 1: The transactional database receives transactional requests for database data from application services and performs transaction operations;

[0039] In the embodiment, the transactional database is responsible for receiving transactional operations of application services on database data, including adding, modifying and deleting, etc. The transactional database is responsible for providing atomicity, consistency, isolation and persistence of application operations, and ensuring strong consistency and reliability requirements for data operations; the specific present invention uses the relational database MySQL for data query, and MySQL is responsible for reliable storage and simple query services for application data. It supports strong consistency and reliability requirements for data operations, including adding, modifying and deleting data, and provides atomicity, consistency, isolation and persistence of application data operations;

[0040] Step 2: The middleware service monitors the database after the transaction operation is executed to further obtain the data operation details; and uses the data operation details to clean and organize the data;

[0041] In the embodiment, after the transaction operation is successfully submitted, the middleware service responsible for monitoring data changes obtains the operation details for the data by monitoring the database operation log or receiving the data operation success message. After receiving the operation details, the middleware service cleans and organizes the received data;

[0042] Step 3: The middleware service encapsulates the cleaned and sorted data to form event messages for consumption by the full-text search engine;

[0043] In the embodiment, after the middleware service receives the data and completes the cleaning and sorting of the data, the data is further encapsulated into an event message for consumption by the full-text search engine, and sent to a high-reliability message queue, so that the full-text search engine can construct a reasonable index data structure based on the received data;

[0044] Step 4: The full-text search engine uses the data operation details to synchronize the data operation to the search engine and build an index structure with hot data sorted in reverse order;

[0045] In the embodiment, the full-text search engine obtains the operation details on the data by consuming the data messages in the message queue, synchronizes the data operations on the transactional database to the search engine, and converts them into operations on the corresponding index data in the search engine;

[0046] Step 5: Set a threshold, and convert data with access frequency and effectiveness below the threshold into cold data; compress and archive the cold data in a scheduled manner and store it in the object storage service; convert data with access frequency and effectiveness above the threshold into hot data;

[0047] Step 6: When the application service performs data query, the query results are obtained according to the query complexity;

[0048] Step 6.1: When the application service performs a hot data query, the query results are obtained through the transactional database and full-text search engine;

[0049] Step 6.2: When the application service performs cold data query, it obtains archived data from the object storage service, decompresses it, creates an index, and performs query;

[0050] In the embodiment, when the application service queries data, it classifies the query according to the complexity. The transactional database's own index can meet the scenario of fast query, and the required results are obtained by directly querying the transactional database. For the query scenario where the transactional database's own index is ineffective, the corresponding results are obtained by querying the full-text search engine;

[0051] A complex retrieval system based on a transactional database big data scenario in this embodiment is used to implement the above method. A complex retrieval system based on a transactional database big data scenario disclosed in the present invention includes a transactional database unit, a data change processing unit, and a data indexing unit;

[0052] The transactional database unit is used to store application data, provide atomicity, consistency, isolation and persistence for data operations, and ensure strong consistency and reliability requirements for data operations; it will serve as the input of the data change processing unit;

[0053] The data change processing unit is used to monitor changes in the database, including addition, update, and deletion operations, and to clean and organize the data;

[0054] The data indexing unit, as the output of the data change processing unit, performs word segmentation on the data content, shields invalid characters, and then constructs an inverted index for subsequent complex retrieval services;

[0055] A computer-readable storage medium of this embodiment is used to store a program of a data transmission method based on a cloud object storage service, and the program of the data transmission method based on the cloud object storage service is used to be executed. The instructions of the data transmission method based on the user datagram protocol are used to implement the complex retrieval method based on a transactional database big data scenario;

[0056] In an embodiment, the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state drive (SSD)).

[0057] An electronic device of this embodiment includes a memory and a processor, wherein the processor executes a program of the data transmission method based on the cloud object storage service to implement the complex retrieval method based on the transactional database big data scenario;

[0058] In an embodiment, an electronic device includes: a memory and a processor; the processor and the memory communicate with each other through a bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the remote procedure call RPC implementation method without IDL of embodiment 1 or any possible implementation thereof. Specifically, the processor can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading the software code stored in the memory. The memory can be integrated in the processor, can be located outside the processor, and exist independently. In an embodiment, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0059] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0060] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A complex retrieval method based on transactional database big data scenario, characterized by: The following steps are included: Step 1: The transactional database receives transactional requests for database data from application services and performs transaction operations; Step 2: The middleware service monitors the database after the transaction operation is executed to further obtain the data operation details; and uses the data operation details to clean and organize the data; Step 3: The middleware service encapsulates the cleaned and sorted data to form event messages for consumption by the full-text search engine; Step 4: The full-text search engine uses the data operation details to synchronize the data operation to the search engine and build an index structure with hot data sorted in reverse order; Step 5: Set a threshold, and convert data with access frequency and effectiveness below the threshold into cold data; compress and archive the cold data in a scheduled manner and store it in the object storage service; convert data with access frequency and effectiveness above the threshold into hot data; Step 6: When the application service performs a data query, the query results are obtained based on the query complexity.

2. A complex retrieval method based on transactional database big data scenario as claimed in claim 1, characterized in that: Step 6 is implemented as follows: Step 6.1: When the application service performs a hot data query, the query results are obtained through the transactional database and full-text search engine; Step 6.2: When the application service performs a cold data query, it obtains the archived data from the object storage service, decompresses it, and creates an index for query.

3. A complex retrieval system based on a transactional database big data scenario that implements the method of claim 1, characterized in that: It includes a transactional database unit, a data change processing unit and a data index unit; The transactional database unit is used to store application data, provide atomicity, consistency, isolation and persistence for data operations, and ensure strong consistency and reliability requirements for data operations; it will serve as the input of the data change processing unit; The data change processing unit is used to monitor changes in the database, including addition, update, and deletion operations, and to clean and organize the data; The data indexing unit, as the output of the data change processing unit, performs word segmentation processing on the data content, shields invalid characters, and then constructs an inverted index for subsequent complex retrieval services.

4. A computer-readable storage medium, wherein the computer-readable storage medium is used to store a program of a data transmission method based on a cloud object storage service, wherein the program of the data transmission method based on the cloud object storage service is used to execute instructions of a data transmission method based on a user datagram protocol to implement the method as described in claim 1.

5. An electronic device comprising a memory and a processor, wherein the processor executes a program of the data transmission method based on the cloud object storage service to implement the method as claimed in claim 1.

Images