Method for creating index data, data query method and device

Abstract

The present application discloses a method for creating index data, a data query method and a device. The method comprises: in the case of monitoring that the flushing status corresponding to the write-ahead log in the database is updated, triggering the log analysis mechanism to take effect, and determining the target data table where the data update currently exists; obtaining pre-stored secondary index annotation information based on the log analysis mechanism, the secondary index annotation information is obtained by the coprocessor parsing the annotation information of the data table in the application during the loading process; using the secondary index annotation information to match the key fields in the write-ahead log to obtain a matching result; in the case where the matching result is used to indicate that the secondary index annotation information is consistent with the key fields in the write-ahead log, using the data content in the write-ahead log to create the secondary index data of the target data table. The entire process of creating index data in the embodiment of the present application is automatically completed by the program, and there is no need to write auxiliary code, which reduces the difficulty of development and enhances stability.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for creating index data, a data query method, and a device. Background Art

[0002] HBase is a popular and mature NoSQL columnar database system in the big data field, capable of high data throughput in massive data environments. Its columnar storage design supports only fast rowkey-based searches. Non-rowkey searches are slower and less efficient. HBase also lacks native support for secondary indexes. Therefore, in field use, business development is difficult without secondary indexes, often requiring the use of auxiliary tables to implement secondary indexes.

[0003] Currently, the most common ways to build secondary indexes are HBase-based co-processing and WAL. In the process of implementing the present invention, the inventors found that the current implementations of these two methods on the market are not automated and often require developers to explicitly program for the secondary index. Summary of the Invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a method for creating index data, a data query method and a device.

[0005] According to one aspect of an embodiment of the present application, a method for creating index data is provided, characterized by comprising:

[0006] When the flush status of the write-ahead log in the database is updated, the log analysis mechanism is triggered to take effect and the target data table with the current data update is determined;

[0007] Acquiring pre-stored secondary index annotation information based on a log analysis mechanism, wherein the secondary index annotation information is obtained by parsing annotation information of a data table in an application during a loading process by a coprocessor;

[0008] Matching the secondary index annotation information with the key fields in the write-ahead log to obtain a matching result;

[0009] When the matching result indicates that the secondary index annotation information is consistent with the key field in the write-ahead log, secondary index data of the target data table is created using the data content in the write-ahead log.

[0010] Furthermore, before monitoring the flush status corresponding to the write-ahead log in the database, the method further includes:

[0011] Obtaining a load instruction triggered by the database, wherein the load instruction is generated by the database after receiving annotation information of a data table sent by a driver deployed on the client, and the annotation information is obtained by the driver scanning an application on the client;

[0012] A loading mechanism is triggered to take effect based on the loading instruction, and the annotation information is parsed using the loading mechanism to obtain the secondary index annotation information.

[0013] Furthermore, the method further comprises:

[0014] A table identifier corresponding to the data table is obtained from the table information of the annotation information, and a message topic corresponding to the data table is registered on a data platform using the table identifier, wherein the data platform is used to receive updated data corresponding to the data table.

[0015] Furthermore, the creating secondary index data of the target data table using the data content in the write-ahead log includes:

[0016] Obtaining data content corresponding to key fields from the write-ahead log;

[0017] Secondary index data of the target data table is created according to a preset index data structure and the data content.

[0018] Furthermore, after creating secondary index data using the data content in the write-ahead log, the method further includes:

[0019] Obtaining a target table identifier corresponding to the target data table;

[0020] Send the secondary index data and the target table identifier to the target message topic in the data platform, so that the data platform updates the secondary index data and the target table identifier to the index library, wherein the target message topic is the message topic corresponding to the target table identifier, and the index library is used to construct a corresponding relationship between the table identifier and the secondary index data.

[0021] According to another aspect of the embodiments of the present application, a data retrieval method is provided, including:

[0022] receiving a data retrieval request, wherein the data retrieval request carries a data table to be queried;

[0023] Obtaining index data association information corresponding to the data table to be queried, and generating a query statement based on the index data association information;

[0024] Sending the query statement to the index library, so that the index library uses the secondary index data in the query statement to feedback a corresponding table identifier;

[0025] The data content corresponding to the data table to be queried is obtained from the database using the table identifier.

[0026] Furthermore, the obtaining of the index data association information corresponding to the data table to be queried includes:

[0027] Detecting whether there is a key field related to the secondary index data in the annotation information corresponding to the data table to be queried;

[0028] In the case where a key field related to the secondary index data exists in the annotation information, the index data association information is constructed based on the data content corresponding to the key field.

[0029] Furthermore, in the case that the key field related to the secondary index data does not exist in the annotation information, the method further includes:

[0030] Query the database for data content corresponding to the data table to be queried.

[0031] According to another aspect of an embodiment of the present application, a device for creating index data is provided, including:

[0032] The monitoring module is used to trigger the log analysis mechanism to take effect and determine the target data table where the data is currently updated when the flush status corresponding to the pre-written log in the database is monitored.

[0033] An acquisition module, configured to acquire pre-stored secondary index annotation information based on a log analysis mechanism, wherein the secondary index annotation information is obtained by the coprocessor parsing target annotation information of a data table in an application during loading;

[0034] a matching module, configured to match the secondary index annotation information with key fields in the write-ahead log to obtain a matching result;

[0035] A creation module is configured to create secondary index data of the target data table using data content in the write-ahead log when the matching result indicates that the secondary index annotation information is consistent with a key field in the write-ahead log.

[0036] According to another aspect of the embodiments of the present application, a data retrieval device is provided, including:

[0037] A receiving module, configured to receive a data retrieval request, wherein the data retrieval request carries a data table to be queried;

[0038] An acquisition module, configured to acquire index data association information corresponding to the data table to be queried, and generate a query statement based on the index data association information;

[0039] A sending module, configured to send the query statement to an index library, so that the index library uses the secondary index data in the query statement to obtain a corresponding table identifier;

[0040] The query module is used to obtain the data content corresponding to the data table to be queried from the database using the table identifier.

[0041] According to another aspect of an embodiment of the present application, a storage medium is further provided, which includes a stored program, and the above steps are executed when the program is run.

[0042] According to another aspect of an embodiment of the present application, an electronic device is also provided, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; wherein: the memory is used to store computer programs; the processor is used to execute the steps in the above method by running the program stored in the memory.

[0043] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the steps in the above method.

[0044] The technical solution provided by the embodiments of the present application has the following advantages over the prior art: The method provided by the embodiments of the present application monitors the flushing status of the write-ahead log through a coprocessor and, when the write-ahead log is updated, matches the key fields in the write-ahead log. It then determines whether it is a secondary index and, if so, automatically creates the secondary index data to enable rapid construction of the secondary index. The entire process is automatically completed by the program, eliminating the need for auxiliary code, reducing development effort and enhancing stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0047] Figure 1 A flowchart of a method for creating index data provided in an embodiment of the present application;

[0048] Figure 2 A flowchart of a data query method provided in another embodiment of the present application;

[0049] Figure 3 A block diagram of a device for creating index data provided in an embodiment of the present application;

[0050] Figure 4 A block diagram of a data query device provided in another embodiment of the present application;

[0051] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0053] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another similar entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0054] The present invention provides a method for creating index data, a data query method, and an apparatus. The method provided by the present invention can be applied to any desired electronic device, such as a server, a terminal, or other electronic device. For ease of description, the method is not specifically limited here and is referred to as an electronic device.

[0055] According to one aspect of the embodiments of the present application, a method embodiment of a method for creating index data is provided. Figure 1 A flowchart of a method for creating index data provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0056] Step S11 : When an update is detected on the flush status corresponding to the write-ahead log in the database, the log analysis mechanism is triggered to take effect, and a target data table with data update is determined.

[0057] In an embodiment of the present application, before monitoring the flush status corresponding to the write-ahead log in the database, the method also includes: obtaining a loading instruction triggered by the database, wherein the loading instruction is generated by the database after receiving annotation information of the data table sent by the driver deployed on the client, and the annotation information is obtained by the driver scanning the client's application; triggering the loading mechanism to take effect based on the loading instruction, and using the loading mechanism to parse the annotation information to obtain the secondary index annotation information.

[0058] It should be noted that the client needs to initiate initialization work before creating index data or updating data. After the driver deployed on the client receives the initialization request initiated by the client, it first scans the application corresponding to the client, obtains the annotation information of each data table, and stores the annotation information. The relevant annotation information will be used in the subsequent query stage. After the driver completes the annotation scan, the driver will request the Hbase database to load the coprocessor. After receiving the request, the Hbase database will load the coprocessor according to the annotation information. Among them, the annotation information includes: @ModelTableAlias ​​is used to describe the database information corresponding to the data to be updated; @ModelPerpertyAlias ​​is used to describe the data field and whether the field is a rowkey (rowkey=true); @ESField is used to describe whether the current field is a secondary index. If it is a secondary index, the current field will be automatically recognized by the coprocessor and finally assembled into index data in ES.

[0059] Based on this, the coprocessor can obtain the loading instructions triggered by the database, and trigger the loading mechanism based on the loading instructions. The coprocessor parses the annotation information during the loading process, obtains the secondary index annotation information, and caches the secondary index annotation information to facilitate the rapid extraction of index data when the data table is updated subsequently.

[0060] In an embodiment of the present application, after loading is completed, the coprocessor will monitor the write status corresponding to the write-ahead log (WAL) in the Hbase database. When the write status corresponding to the write-ahead log in the Hbase database is updated, the log analysis mechanism is triggered to take effect and the target data table with data update is determined. Specifically, the client initiates a data update request through the driver (Driver). The driver (Driver) sends the data to the Hbase database through the interface of the Hbase database for data update. The Hbase database first writes the updated data to the write-ahead log. At this time, the coprocessor will monitor the update of the write status corresponding to the write-ahead log (WAL).

[0061] Step S12: obtaining pre-stored secondary index annotation information based on the log analysis mechanism, wherein the secondary index annotation information is obtained by the coprocessor parsing the annotation information of the data table in the application during the loading process.

[0062] In an embodiment of the present application, since the coprocessor caches the secondary index annotation information obtained by parsing the annotation information of the data table during the loading process, after the log analysis mechanism is triggered to take effect, the pre-stored secondary index annotation information can be directly obtained based on the log analysis mechanism.

[0063] Step S13: Use the secondary index annotation information to match the key fields in the write-ahead log to obtain a matching result.

[0064] In an embodiment of the present application, the coprocessor uses the secondary index annotation information to extract the data content of the key field (i.e., @ESField.name) in the write-ahead log for matching. If the data content described by @ESField.name is equal to the data content extracted from the WAL, it means that the data content is a secondary index. In this case, the matching result is a successful match, indicating that the secondary index annotation information is consistent with the key field in the write-ahead log. On the contrary, if the data content described by @ESField.name is not equal to the data content extracted from the WAL, it means that the data content is not a secondary index. In this case, the matching result is a failed match, indicating that the secondary index annotation information is inconsistent with the key field in the write-ahead log.

[0065] Step S14 : When the matching result indicates that the secondary index annotation information is consistent with the key field in the write-ahead log, secondary index data of the target data table is created using the data content in the write-ahead log.

[0066] In an embodiment of the present application, secondary index data of a target data table is created using the data content in the write-ahead log, including: obtaining data content corresponding to key fields from the write-ahead log; and creating secondary index data of the target data table according to a preset index data structure and data content.

[0067] In an embodiment of the present application, in the process of creating secondary index data, the data content corresponding to the key field is first obtained from the pre-written log, and the preset index data structure is obtained, and the data content is written into the preset index data structure to obtain the secondary index data of the target data table.

[0068] The method provided in the embodiments of the present application uses a coprocessor to monitor the flushing status of the write-ahead log and, if the write-ahead log is updated, matches the key fields in the write-ahead log. It then determines whether it is a secondary index and, if so, automatically creates the secondary index data to enable rapid construction of the secondary index. The entire process is automatically completed by the program, eliminating the need for auxiliary code, reducing development effort and enhancing stability.

[0069] In an embodiment of the present application, after creating secondary index data using the data content in the write-ahead log, the method also includes: obtaining a target table identifier corresponding to the target data table; sending the secondary index data and the target table identifier to a target message topic in the data platform, so that the data platform updates the secondary index data and the target table identifier to the index library, wherein the target message topic is a message topic corresponding to the target table identifier, and the index library is used to construct a correspondence between the table identifier and the secondary index data.

[0070] Specifically, after the coprocessor creates the secondary index data, it sends the secondary index data to the target message topic in Kafka using the data platform (Kafka) interface according to the target table identifier described by @ModelTableAlias.tableName as the topic. One message topic corresponds to one data table. After receiving the data, Kafka automatically pushes the secondary index data and the target table identifier to Spark-Streaming according to the subscription logic. At this time, Spark-Streaming completes the initialization work, and Spark-Streaming receives the secondary index data and the target table identifier. Spark-Streaming receives Kafka and reassembles it according to the ES data structure and secondary index data to obtain the final secondary index data. Spark-Streaming sends the final secondary index data created to the index library so that the index library can build a correspondence between the table identifier and the secondary index data.

[0071] It should be noted that during the initialization of Kafka, the coprocessor obtains the table identifier corresponding to the data table from the table information in the annotation information, and uses the table identifier to register the message topic corresponding to the data table on the data platform, so that when the data in the subsequent data table is updated, the updated data will be sent to the message topic.

[0072] The method provided in this embodiment uses a coprocessor combined with Kafka to separate the processes of monitoring data changes and creating indexes. This minimizes the impact of index creation on HBase database query speeds while also ensuring data is not lost. Furthermore, the Spark Streaming-based distributed stream computing engine enables rapid index building, minimizing latency in the index creation process.

[0073] Figure 2 A flowchart of a data retrieval method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method may include the following steps:

[0074] Step S21: receiving a data retrieval request, wherein the data retrieval request carries a data table to be queried.

[0075] Step S22: obtaining index data association information corresponding to the data table to be queried, and generating a query statement based on the index data association information.

[0076] In an embodiment of the present application, obtaining index data association information corresponding to the data table to be queried includes: detecting whether there are key fields related to the secondary index data in the annotation information corresponding to the data table to be queried; and when there are key fields related to the secondary index data in the annotation information, constructing index data association information based on the data content corresponding to the key fields.

[0077] Step S23: Send the query statement to the index library, so that the index library uses the secondary index data in the query statement to feed back a corresponding table identifier.

[0078] Step S24: using the table identifier, obtain the data content corresponding to the data table to be queried from the database.

[0079] In the embodiment of the present application, when an application triggers a data retrieval request, it is not necessary to describe the secondary index information in the data retrieval request, but only to indicate the data table to be queried. The subsequent driver automatically initiates the retrieval based on the annotation of the data table to be queried.

[0080] Based on this, after receiving a data retrieval request, the driver first obtains the index data association information from the annotation information of the data table to be queried in the application. If there is a key field used to describe the secondary index data in this query, a query statement with the key field and the data content corresponding to the key field as conditions is automatically constructed, thereby realizing the automation of the retrieval. Secondly, the driver uses the query statement to search the index library to obtain the retrieval content. Specifically, the data content and key fields in the query statement are used to query the matching target secondary index data. At the same time, because the association relationship between the table identifier and the secondary index data has been created in the index library during the index creation process, the target secondary index data can be directly used to query the corresponding table identifier. Finally, after obtaining the table identifier, the driver uses the table identifier to quickly retrieve information from the Hbase database. After the retrieval is completed, the data results are returned to the application.

[0081] In an embodiment of the present application, when there is no key field related to the secondary index data in the annotation information, the method further includes: querying the database for data content corresponding to the data table to be queried.

[0082] The data retrieval method provided in the embodiment of the present application retrieves data in Hbase. If there is a secondary index in the annotation, the query process will automatically give priority to the secondary index data, thereby adopting the traditional SQL mode design in the data modeling and data implementation process to improve design and development efficiency.

[0083] Figure 3 This is a block diagram of a device for creating index data provided by an embodiment of the present application. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 3 As shown, the device includes:

[0084] The monitoring module 31 is used to trigger the log analysis mechanism to take effect and determine the target data table where the data is currently updated when the flush status corresponding to the write-ahead log in the database is monitored to be updated;

[0085] An acquisition module 32 is configured to acquire pre-stored secondary index annotation information based on a log analysis mechanism, wherein the secondary index annotation information is obtained by parsing target annotation information of a data table in an application program during loading by the coprocessor;

[0086] A matching module 33 is configured to match the secondary index annotation information with the key fields in the write-ahead log to obtain a matching result;

[0087] The creation module 34 is configured to create secondary index data of the target data table using the data content in the write-ahead log when the matching result indicates that the secondary index annotation information is consistent with the key field in the write-ahead log.

[0088] In an embodiment of the present application, the index data creation device also includes: a loading module, which is used to obtain a loading instruction triggered by a database, wherein the loading instruction is generated by the database after receiving annotation information of a data table sent by a driver deployed on the client, and the annotation information is obtained by the driver scanning the client's application; the loading mechanism is triggered to take effect based on the loading instruction, and the annotation information is parsed using the loading mechanism to obtain secondary index annotation information.

[0089] In an embodiment of the present application, the index data creation device also includes: a registration module, which is used to obtain the table identifier corresponding to the data table from the table information of the annotation information, and use the table identifier to register the message topic corresponding to the data table on the data platform, wherein the data platform is used to receive the updated data corresponding to the data table.

[0090] In an embodiment of the present application, a creation module is used to obtain data content corresponding to key fields from a write-ahead log; and generate secondary index data for creating a target data table according to a preset index data structure and data content.

[0091] In an embodiment of the present application, the index data creation device also includes: a processing module for obtaining a target table identifier corresponding to the target data table; sending secondary index data and the target table identifier to a target message topic in the data platform, so that the data platform updates the secondary index data and the target table identifier to the index library, wherein the target message topic is a message topic corresponding to the target table identifier, and the index library is used to construct a correspondence between the table identifier and the secondary index data.

[0092] Figure 4 This is a block diagram of a data retrieval device provided in an embodiment of the present application. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 4 As shown, the device includes:

[0093] The receiving module 41 is configured to receive a data retrieval request, wherein the data retrieval request carries a data table to be queried;

[0094] The acquisition module 42 is used to obtain the secondary index data corresponding to the data table to be queried and generate a query statement based on the secondary index data;

[0095] The sending module 43 is used to send the query statement to the index library, so that the index library uses the secondary index data in the query statement to obtain the corresponding table identifier;

[0096] The query module 44 is used to obtain the data content corresponding to the data table to be queried from the database using the table identifier.

[0097] In an embodiment of the present application, the acquisition module 42 is used to detect whether there are key fields related to the secondary index data in the annotation information corresponding to the data table to be queried; when there are key fields related to the secondary index data in the annotation information, the secondary index data is constructed based on the data content corresponding to the key fields.

[0098] In an embodiment of the present application, the data retrieval device Haibo Aka: a processing module, is used to query the data content corresponding to the data table to be queried from the database.

[0099] The present application also provides an electronic device, such as Figure 5 As shown, the electronic device may include: a processor 1501 , a communication interface 1502 , a memory 1503 and a communication bus 1504 , wherein the processor 1501 , the communication interface 1502 , and the memory 1503 communicate with each other via the communication bus 1504 .

[0100] Memory 1503, used for storing computer programs;

[0101] The processor 1501 is configured to implement the steps of the above embodiment when executing the computer program stored in the memory 1503 .

[0102] The communication bus mentioned in the terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0103] The communication interface is used for communication between the above terminal and other devices.

[0104] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0105] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0106] In another embodiment provided by the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute any of the methods described in the above embodiments.

[0107] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the methods described in the above embodiments.

[0108] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive SolidState Disk).

[0109] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the scope of protection of the present application.

[0110] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

Claims

1. A method for creating index data, characterized in that: include: When the flush status of the write-ahead log in the HBase database is updated, the log analysis mechanism is triggered to take effect and the target data table with the current data update is determined; Acquiring pre-stored secondary index annotation information based on a log analysis mechanism, wherein the secondary index annotation information is obtained by parsing annotation information of a data table in an application during a loading process by a coprocessor; Matching the secondary index annotation information with the key fields in the write-ahead log to obtain a matching result; When the matching result indicates that the secondary index annotation information is consistent with the key field in the write-ahead log, creating secondary index data of the target data table using the data content in the write-ahead log; Before monitoring the flush status corresponding to the write-ahead log in the HBase database, the method further includes: Obtaining a load instruction triggered by the database, wherein the load instruction is generated by the database after receiving annotation information of a data table sent by a driver deployed on the client, and the annotation information is obtained by the driver scanning an application on the client; A loading mechanism is triggered to take effect based on the loading instruction, and the annotation information is parsed using the loading mechanism to obtain the secondary index annotation information.

2. The method according to claim 1, characterized in that The method further comprises: A table identifier corresponding to the data table is obtained from the table information of the annotation information, and a message topic corresponding to the data table is registered on a data platform using the table identifier, wherein the data platform is used to receive updated data corresponding to the data table.

3. The method according to claim 1, characterized in that The step of creating secondary index data of the target data table by using the data content in the write-ahead log includes: Obtaining data content corresponding to key fields from the write-ahead log; Secondary index data of the target data table is created according to a preset index data structure and the data content.

4. The method according to claim 1, wherein After creating secondary index data using the data content in the write-ahead log, the method further includes: Obtaining a target table identifier corresponding to the target data table; Send the secondary index data and the target table identifier to the target message topic in the data platform, so that the data platform updates the secondary index data and the target table identifier to the index library, wherein the target message topic is the message topic corresponding to the target table identifier, and the index library is used to construct a corresponding relationship between the table identifier and the secondary index data.

5. A data retrieval method, characterized in that: include: receiving a data retrieval request, wherein the data retrieval request carries a data table to be queried; Obtaining index data association information corresponding to the data table to be queried, and generating a query statement based on the index data association information; Sending the query statement to the index library so that the index library uses the secondary index data in the query statement to feedback the corresponding table identifier, wherein the index library is used to establish a correspondence between the table identifier and the secondary index data, and the secondary index is created using the data content in the write-ahead log, including: triggering the log analysis mechanism to take effect when monitoring an update of the flush status corresponding to the write-ahead log in the HBase database, and determining the target data table where the data is currently updated; Acquiring pre-stored secondary index annotation information based on a log analysis mechanism, wherein the secondary index annotation information is obtained by parsing annotation information of a data table in an application during a loading process by a coprocessor; Matching the secondary index annotation information with the key fields in the write-ahead log to obtain a matching result; When the matching result indicates that the secondary index annotation information is consistent with the key field in the write-ahead log, creating secondary index data of the target data table using the data content in the write-ahead log; Before monitoring the flush status corresponding to the write-ahead log in the HBase database, the method further includes: Obtaining a load instruction triggered by the database, wherein the load instruction is generated by the database after receiving annotation information of a data table sent by a driver deployed on the client, and the annotation information is obtained by the driver scanning an application on the client; triggering a loading mechanism to take effect based on the loading instruction, and parsing the annotation information using the loading mechanism to obtain the secondary index annotation information; Using the table identifier, obtain the data content corresponding to the data table to be queried from the Hbase database; The obtaining of the index data association information corresponding to the data table to be queried includes: detecting whether there is a key field related to the secondary index data in the annotation information corresponding to the data table to be queried; and when there is a key field related to the secondary index data in the annotation information, constructing the index data association information based on the data content corresponding to the key field.

6. The method according to claim 5, characterized in that In a case where the key field related to the secondary index data does not exist in the annotation information, the method further includes: querying the Hbase database for data content corresponding to the data table to be queried.

7. A device for creating index data, characterized in that: include: The monitoring module is used to trigger the log analysis mechanism to take effect and determine the target data table where data is currently updated when the flush status corresponding to the pre-written log in the HBase database is updated; An acquisition module, configured to acquire pre-stored secondary index annotation information based on a log analysis mechanism, wherein the secondary index annotation information is obtained by the coprocessor parsing target annotation information of a data table in an application during loading; a matching module, configured to match the secondary index annotation information with key fields in the write-ahead log to obtain a matching result; a creation module configured to create secondary index data of the target data table using data content in the write-ahead log when the matching result indicates that the secondary index annotation information is consistent with a key field in the write-ahead log; The index data creation device also includes: a loading module, which is used to obtain a loading instruction triggered by the database before monitoring the flush status corresponding to the pre-written log in the Hbase database, wherein the loading instruction is generated by the database after receiving the annotation information of the data table sent by the driver deployed on the client, and the annotation information is obtained by the driver scanning the client application; based on the loading instruction, the loading mechanism is triggered to take effect, and the annotation information is parsed using the loading mechanism to obtain the secondary index annotation information.

8. A data retrieval device, characterized in that: include: A receiving module, configured to receive a data retrieval request, wherein the data retrieval request carries a data table to be queried; An acquisition module, configured to acquire index data association information corresponding to the data table to be queried, and generate a query statement based on the index data association information; A sending module, for sending the query statement to the index library, so that the index library obtains the corresponding table identifier using the secondary index data in the query statement, wherein the index library is used to build a corresponding relationship between the table identifier and the secondary index data, and the secondary index is created using the data content in the write-ahead log, including: when monitoring that the flush status corresponding to the write-ahead log in the Hbase database is updated, triggering the log analysis mechanism to take effect, and determining the target data table where data updates are currently available; obtaining pre-stored secondary index annotation information based on the log analysis mechanism, wherein the secondary index annotation information is obtained by parsing the annotation information of the data table in the application program during the loading process by the coprocessor; using the secondary index annotation information to match the key field in the write-ahead log to obtain a matching result; when the matching result indicates that the secondary index annotation information is consistent with the key field in the write-ahead log, using the data content in the write-ahead log to create the secondary index data of the target data table; Before monitoring the flush status of the write-ahead log in the HBase database, the following steps are also required: Obtaining a load instruction triggered by the database, wherein the load instruction is generated by the database after receiving annotation information of a data table sent by a driver deployed on the client, and the annotation information is obtained by the driver scanning an application on the client; triggering a loading mechanism to take effect based on the loading instruction, and parsing the annotation information using the loading mechanism to obtain the secondary index annotation information; A query module is used to obtain the data content corresponding to the data table to be queried from the Hbase database using the table identifier; The acquisition module is used to detect whether there are key fields related to the secondary index data in the annotation information corresponding to the data table to be queried; when there are key fields related to the secondary index data in the annotation information, the index data association information is constructed based on the data content corresponding to the key fields.

Images