A Low-Latency Materialized View Approach

Abstract

This invention discloses a low-latency materialized view method, comprising the following steps: S1, after the application server initiates a database transaction to the source database and executes data operation instructions, before the database transaction is committed, a state intent event is constructed. This relates to the field of database technology. This application uses heavy data change information as the state intent event and a lightweight commit confirmation signal as the commit state event. By separating the state intent event and the commit state event and processing them asynchronously in parallel, the update latency of the materialized view no longer depends on the entire CDC link, but only on the transmission latency of the commit confirmation signal. This reduces transmission latency, achieves lower-latency data synchronization, and alleviates the high latency problem inherent in traditional materialized view updates that rely on change data capture technology.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a low-latency synchronization method for multi-table related data to materialized view data in a distributed database system. Background Technology

[0002] In the field of database technology, materialized views are a key database object whose core value lies in pre-calculating and storing the results of complex queries, thereby significantly shortening the response time of subsequent similar queries. This provides efficient data access support for complex database query scenarios and is widely used in data analysis, business report generation, and other business scenarios. A materialized view is a database object that includes the results of a query; it can be a local copy of remote data or used to generate summary tables based on the summation of data tables.

[0003] Currently, there are the following issues with updating materialized views:

[0004] Currently, materialized view updates typically rely on Change Data Capture (CDC) technology. The traditional CDC workflow involves multiple steps: first, waiting for the source database to asynchronously flush its transaction logs to disk to ensure data persistence; then, a dedicated component reads and parses the log files to extract committed data changes; next, the extracted change data is serialized and transmitted over the network to a stream processing engine; finally, the stream processing engine performs data computation and integration before updating the materialized view with the final result. CDC technology asynchronously reads the database's transaction logs to obtain committed data changes, then updates the materialized view after computation by the stream processing engine. However, CDC technology inherently suffers from high latency, accumulated from log flushing, log parsing, data serialization, network transmission, and stream processing computation, making it difficult to meet the real-time requirements of applications. Summary of the Invention

[0005] To address the technical problems existing in the background art, this invention proposes a materialized view method with low data latency.

[0006] This invention proposes a low-latency materialized view method, applicable to a system including an application server, a source database, a message middleware, a materialized view consumption service, and a materialized view storage. The method includes the following steps:

[0007] S1. After the application server initiates a database transaction to the source database and executes data operation instructions, and before the database transaction is committed, construct a state intent event.

[0008] S2. The application server performs an asynchronous publishing operation for the state intent event, sending the state intent event to the message middleware; the message middleware is used to receive and temporarily store the state intent event.

[0009] As an explanation, the purpose of setting up a message middleware is twofold: firstly, the application server can immediately continue other work after handing over the state intent event to the message middleware without waiting for subsequent processing to complete; secondly, the message middleware can ensure that the state intent event is not lost due to network fluctuations or temporary service failures in the application server, while allowing the application server and the materialized view consumer service to operate and maintain independently without having to establish a direct connection.

[0010] S3. Construct an in-memory overlay state table in the materialized view consumption service. The materialized view consumption service reads state intent events from the message middleware and generates overlay state entries in the in-memory overlay state table based on the state intent events.

[0011] S4. After the database transaction is successfully committed, a commit status event is constructed in the source database and transmitted to the materialized view consumption service.

[0012] S5. After receiving the commit status event, the materialized view consumption service generates a final overlay status entry in the in-memory overlay status table and updates the new status value after data operation in the final overlay status entry to the corresponding materialized view record in the materialized view storage.

[0013] Preferably, in S1, the state intent event is constructed as follows:

[0014] Extract a unique transaction identifier from a database transaction;

[0015] For illustration, a unique transaction identifier is, for example, the TxID assigned to the transaction by the source database;

[0016] TxID stands for Transaction ID, which refers to a transaction identifier, a unique numeric or string identifier assigned by the source database to each initiated database transaction;

[0017] Extract the primary key and data operation type of the data being manipulated from the data manipulation instructions;

[0018] As an explanation, the primary key of the data being operated on is such as order_id in the order table and user_id in the user table. The data operation types are such as INSERT, UPDATE, and DELETE. Here, order_id refers to the order number, user_id refers to the user number, and INSERT, UPDATE, and DELETE are the three most basic data modification actions in the database, corresponding to the three business scenarios of adding, modifying, and deleting.

[0019] Obtain the generated information after the data operation instruction is executed. The generated information includes the new status value after the data operation.

[0020] A distributed, unique, ordered ID generation algorithm is used to generate a version number after data operations on the same data. The version number after data operations is used to identify the order of data operations.

[0021] As an explanation, the version number generated by a later data operation is higher; the version number after a data operation is used to identify the order of data operations, which refers to the global order of all data operations on the same data from different data operation instructions.

[0022] The unique transaction identifier, the primary key of the data being operated on, the data operation type, the new status value after the data operation, and the version number after the data operation are encapsulated together into a JSON format status intent event.

[0023] For illustration, JSON is an existing lightweight data exchange format.

[0024] Preferably, in S3, a memory overlay state table is constructed in the materialized view consumption service, as follows:

[0025] When the materialized view consumption service starts, a key-value pair data structure based on the primary key index is automatically initialized in the memory space of the materialized view consumption service as the basic architecture of the memory overlay state table.

[0026] Using the primary key of the data being operated on as the unique index key, a hash index is used to generate a mapping rule between the primary key of the data being operated on and the superimposed state entries, thus generating a memory superimposed state table.

[0027] Preferably, in S3, based on the state intent event, a superimposed state entry is generated in the memory superimposed state table, as follows:

[0028] Generate a corresponding overlay state entry for the primary key of the data being manipulated in each state intent event;

[0029] The superimposed status entries include the primary key of the data being operated on, the unique transaction identifier, the new status value after the data operation, the version number after the data operation, and the status identifier that indicates the validity of the entry.

[0030] Among them, the status of the entry is marked as pending;

[0031] When the primary key of the same manipulated data corresponds to multiple state intent events, and the primary key of the same manipulated data has an overlay state entry in the memory overlay state table, the materialized view consumption service compares the version number of the newly received state intent event after data operation with the version number of the overlay state entry after data operation, and retains the overlay state entry corresponding to the state intent event with the highest version number after data operation.

[0032] Preferably, in S4, a commit status event is constructed in the source database as follows:

[0033] After a database transaction is successfully committed, the transaction status is marked as successfully committed in the source database.

[0034] Extract the unique transaction identifier of the database transaction, and encapsulate the unique transaction identifier and the successful commit status in a preset format to form a commit status event.

[0035] Preferably, in S4, the commit status event is transmitted to the materialized view consumption service through a lightweight message channel, which is either the pg_notify mechanism of the PostgreSQL database or the BINLOG EVENT notification mechanism of the MySQL database.

[0036] As an explanation, PostgreSQL is an existing object-relational database management system, and the pg_notify mechanism is a native publish-subscribe notification feature provided by the existing PostgreSQL database.

[0037] MySQL is an existing relational database management system, and the BINLOG EVENT notification mechanism is a native publish-subscribe notification feature provided by the existing MySQL database.

[0038] Preferably, in step S5, after receiving the commit status event, the materialized view consumption service generates a final overlay status entry in the in-memory overlay status table, as follows:

[0039] Based on the transaction identifier in the commit status event, search for all overlay status entries containing that transaction identifier in the memory overlay status table; for each overlay status entry found, update the status identifier of the overlay status entry from pending to committed.

[0040] For all committed state entries corresponding to the primary key of the same manipulated data, select the state entry with the highest version number after the data operation as the final state entry.

[0041] A low-latency materialized view system, comprising:

[0042] Application server, source database, message middleware, materialized view consumption service, and materialized view storage, among which:

[0043] Application server: Used to initiate database transactions to the source database and execute data operation instructions. After executing the data operation instructions, it constructs a state intent event before the database transaction is committed. It is used to perform asynchronous publishing operations of the state intent event and send the state intent event to the message middleware.

[0044] Message middleware: Used to receive and temporarily store state intent events sent by the application server, so that the materialized view consumption service can read the state intent events, and to decouple the application server from the materialized view consumption service, ensuring that the state intent events are not lost due to network fluctuations or temporary service failures of the application server.

[0045] Source database: Used to mark the status of a database transaction as successfully committed after the database transaction is successfully committed. It constructs a commit status event based on the unique transaction identifier of the database transaction and the successful commit status, and transmits the commit status event to the materialized view consumption service through a lightweight message channel.

[0046] Materialized View Consumption Service: Used to build an in-memory overlay state table, read state intent events from the message middleware, generate overlay state entries in the in-memory overlay state table based on the state intent events, receive commit state events transmitted from the source database, update the state flags that identify the validity of overlay state entries based on the commit state events, and filter out the final overlay state entries.

[0047] Materialized view storage: Used to receive the final overlay status entry sent by the materialized view consumption service, and update the corresponding materialized view record with the new status value after the data operation in the final overlay status entry.

[0048] The materialized view method with low data latency proposed in this invention has the following beneficial technical effects:

[0049] 1. This application treats weight data change information as a state intent event and lightweight commit confirmation signal as a commit state event. By separating the state intent event and the commit state event and processing them asynchronously in parallel, the update latency of the materialized view no longer depends on the entire CDC link, but only on the transmission latency of the commit confirmation signal. This reduces transmission latency, achieves lower latency data synchronization, and alleviates the inherent high latency problem of traditional materialized view updates relying on change data capture technology.

[0050] 2. The in-memory overlay state table in this application generates version numbers for data operations using a distributed unique ordered ID generation algorithm. Different data operation instructions for the same data retain the state intent event with the highest version number after the data operation. Simultaneously, it combines the status identifier of the commit status event to update the validity of the entry, ensuring the accurate global sequence of data operations and avoiding materialized view data errors caused by disordered operation timing. The in-memory overlay state table uses the primary key of the operated data as the unique index key and employs a hash index to generate a mapping rule between the primary key of the operated data and overlay state entries. This allows for the location of related overlay state entries with the same primary key or unique transaction identifier for the operated data. After receiving a commit status event, only overlay state entries with the highest version number after the data operation that have been committed are selected as the final overlay state entries updated to the materialized view. This effectively eliminates interference from invalid data such as uncommitted or low version numbers after data operations, ensuring that the materialized view records in the materialized view storage are consistent with the effective changes in the source database, thus improving the accuracy of the materialized view data. Attached Figure Description

[0051] Figure 1 This is a flowchart of a low-latency materialized view method according to the present invention;

[0052] Figure 2 This is a schematic diagram of a low-latency materialized view system according to the present invention. Detailed Implementation

[0053] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0054] like Figure 1 The illustrated materialized view method, applied to a system including an application server, a source database, a message middleware, a materialized view consumption service, and a materialized view storage, includes the following steps:

[0055] S1. After the application server initiates a database transaction to the source database and executes data operation instructions, and before the database transaction is committed, construct a state intent event.

[0056] For illustrative purposes, the application server is the server used in this application;

[0057] S2. The application server performs an asynchronous publishing operation for the state intent event, sending the state intent event to the message middleware; the message middleware is used to receive and temporarily store the state intent event.

[0058] The goal is twofold: firstly, the application server can immediately continue other tasks after handing over the state intent event to the message middleware without waiting for subsequent processing to complete; secondly, the message middleware can ensure that the state intent event is not lost due to network fluctuations or temporary service failures in the application server, while allowing the application server and the materialized view consumption service to operate and maintain independently without having to establish a direct connection.

[0059] S3. Construct an in-memory overlay state table in the materialized view consumption service. The materialized view consumption service reads state intent events from the message middleware and generates overlay state entries in the in-memory overlay state table based on the state intent events.

[0060] S4. After the database transaction is successfully committed, a commit status event is constructed in the source database and transmitted to the materialized view consumption service.

[0061] S5. After receiving the commit status event, the materialized view consumption service generates a final overlay status entry in the in-memory overlay status table and updates the new status value after data operation in the final overlay status entry to the corresponding materialized view record in the materialized view storage.

[0062] This application separates the weight data change information as a state intent event and the lightweight commit confirmation signal as a commit state event, and processes the state intent event and commit state event asynchronously and in parallel. This makes the update latency of the materialized view no longer depend on the entire CDC link, but only on the transmission latency of the commit confirmation signal, thereby reducing transmission latency and achieving lower latency data synchronization. Moreover, this application does not intrude on the database kernel, has good versatility and feasibility, and alleviates the high latency problem of traditional materialized view updates relying on change data capture technology.

[0063] In an optional embodiment, in S1, a state intent event is constructed as follows:

[0064] Extract a unique transaction identifier from a database transaction;

[0065] For illustration, a unique transaction identifier is, for example, the TxID assigned to the transaction by the source database;

[0066] TxID stands for Transaction ID, which refers to a transaction identifier, a unique numeric or string identifier assigned by the source database to each initiated database transaction;

[0067] Extract the primary key and data operation type of the data being manipulated from the data manipulation instructions;

[0068] As an explanation, the primary key of the data being operated on is such as order_id in the order table and user_id in the user table. The data operation types are such as INSERT, UPDATE, and DELETE. Here, order_id refers to the order number, user_id refers to the user number, and INSERT, UPDATE, and DELETE are the three most basic data modification actions in the database, corresponding to the three business scenarios of adding, modifying, and deleting.

[0069] Obtain the generated information after the data operation instruction is executed. The generated information includes the new status value after the data operation.

[0070] A distributed, unique, ordered ID generation algorithm is used to generate a version number after data operations on the same data. The version number after data operations is used to identify the order of data operations.

[0071] In one optional embodiment, the distributed unique ordered ID generation algorithm is the snowflake algorithm;

[0072] As an explanation, the version number generated by a later data operation is higher; the version number after a data operation is used to identify the order of data operations, which refers to the global order of all data operations on the same data from different data operation instructions.

[0073] The unique transaction identifier, the primary key of the data being operated on, the data operation type, the new status value after the data operation, and the version number after the data operation are encapsulated together into a JSON format status intent event.

[0074] For illustration, JSON is an existing lightweight data exchange format;

[0075] In an optional embodiment, in S3, a memory overlay state table is constructed in the materialized view consumption service as follows:

[0076] When the materialized view consumption service starts, a key-value pair data structure based on the primary key index is automatically initialized in the memory space of the materialized view consumption service as the basic architecture of the memory overlay state table.

[0077] Using the primary key of the data being operated on as the unique index key, a hash index is used to generate a mapping rule between the primary key of the data being operated on and the superimposed state entries, thus generating an in-memory superimposed state table.

[0078] In an optional embodiment, in S3, based on the state intent event, an overlay state entry is generated in the memory overlay state table as follows:

[0079] Generate a corresponding overlay state entry for the primary key of the data being manipulated in each state intent event;

[0080] The superimposed status entries include the primary key of the data being operated on, the unique transaction identifier, the new status value after the data operation, the version number after the data operation, and the status identifier that indicates the validity of the entry.

[0081] Among them, the status of the entry is marked as pending;

[0082] When the primary key of the same manipulated data corresponds to multiple state intent events, and the primary key of the same manipulated data has an overlay state entry in the memory overlay state table, the materialized view consumption service will compare the version number of the newly received state intent event after data operation with the version number of the overlay state entry after data operation, and retain the overlay state entry corresponding to the state intent event with the highest version number after data operation.

[0083] In an optional embodiment, in S4, a commit status event is constructed in the source database as follows:

[0084] After a database transaction is successfully committed, the transaction status is marked as successfully committed in the source database.

[0085] Extract the unique transaction identifier of the database transaction, and encapsulate the unique transaction identifier and the successful commit status of the database transaction in a preset format to form a commit status event;

[0086] In an optional embodiment, in S4, the commit status event is transmitted to the materialized view consumption service via a lightweight message channel, which is either the pg_notify mechanism of the PostgreSQL database or the BINLOG EVENT notification mechanism of the MySQL database.

[0087] As an explanation, PostgreSQL is an existing object-relational database management system, and the pg_notify mechanism is a native publish-subscribe notification feature provided by the existing PostgreSQL database.

[0088] MySQL is an existing relational database management system, and the BINLOG EVENT notification mechanism is a native publish-subscribe notification feature provided by the existing MySQL database.

[0089] In an optional embodiment, in S5, after receiving the commit status event, the materialized view consumption service generates a final overlay status entry in the in-memory overlay status table, as follows:

[0090] Based on the transaction identifier in the commit status event, search for all overlay status entries containing that transaction identifier in the memory overlay status table; for each overlay status entry found, update the status identifier of the overlay status entry from pending to committed.

[0091] For all committed state entries corresponding to the primary key of the same manipulated data, select the state entry with the highest version number after the data operation as the final state entry.

[0092] The in-memory overlay state table in this application generates version numbers for data operations using a distributed unique ordered ID generation algorithm. Different data operation instructions for the same data retain the state intent event with the highest version number after the data operation. Simultaneously, it combines the status identifier of the commit status event to update the validity of the entry, ensuring the accurate global sequence of data operations and avoiding materialized view data errors caused by disordered operation timing. The in-memory overlay state table uses the primary key of the operated data as the unique index key and employs a hash index to generate a mapping rule between the primary key of the operated data and overlay state entries. This allows for the location of related overlay state entries with the same primary key or unique transaction identifier for the operated data. After receiving a commit status event, only overlay state entries with the highest version number after the data operation that have been committed are selected as the final overlay state entries updated to the materialized view. This effectively eliminates interference from invalid data such as uncommitted or low version numbers after data operations, ensuring that the materialized view records in the materialized view storage are consistent with the effective changes in the source database, thus improving the accuracy of the materialized view data.

[0093] like Figure 2 The illustrated materialized view system with low data latency includes:

[0094] Application server, source database, message middleware, materialized view consumption service, and materialized view storage, among which:

[0095] Application server: Used to initiate database transactions to the source database and execute data operation instructions. After executing the data operation instructions, it constructs a state intent event before the database transaction is committed. It is used to perform asynchronous publishing operations of the state intent event and send the state intent event to the message middleware.

[0096] Message middleware: Used to receive and temporarily store state intent events sent by the application server, so that the materialized view consumption service can read the state intent events, and to decouple the application server from the materialized view consumption service, ensuring that the state intent events are not lost due to network fluctuations or temporary service failures of the application server.

[0097] Source database: Used to mark the status of a database transaction as successfully committed after the database transaction is successfully committed. It constructs a commit status event based on the unique transaction identifier of the database transaction and the successful commit status, and transmits the commit status event to the materialized view consumption service through a lightweight message channel.

[0098] Materialized View Consumption Service: Used to build an in-memory overlay state table, read state intent events from the message middleware, generate overlay state entries in the in-memory overlay state table based on the state intent events, receive commit state events transmitted from the source database, update the state flags that identify the validity of overlay state entries based on the commit state events, and filter out the final overlay state entries.

[0099] Materialized view storage: Used to receive the final overlay status entry sent by the materialized view consumption service, and update the corresponding materialized view record with the new status value after the data operation in the final overlay status entry.

[0100] Specifically, the workflow for consuming materialized views includes the following steps:

[0101] S401: Start;

[0102] S402: Read a message from the message broker;

[0103] S403: Determine if the message is a state intent event. If yes, proceed to S404; otherwise, proceed to S408.

[0104] S404: Parse the state intent event and extract the primary key, unique transaction identifier, new state value after data operation, and version number after data operation of the operated data;

[0105] S405: In the memory overlay status table, check whether the primary key of the operated data already has an overlay status entry;

[0106] S406: If it does not exist, create a new overlay status entry for the primary key of the operated data in the memory overlay status table, record the unique transaction identifier, the new status value after the data operation, the version number after the data operation, and mark the status that indicates the validity of the entry as pending.

[0107] S407: If it already exists, compare the version number of the newly received data operation with the version number of the existing overlay status entry; if the version number of the newly received data operation is higher, update the unique transaction identifier, the new status value after the data operation, and the version number after the data operation of the overlay status entry, and keep the status identifier of the entry's validity in the pending state; after completion, return to S402.

[0108] S408: Determine if the message is a commit status event. If yes, proceed to S409; otherwise, return to S402.

[0109] S409: Parse the commit status event to extract the unique transaction identifier and commit success status of the database transaction;

[0110] S410: In the memory overlay state table, find all overlay state entries associated with this unique transaction identifier;

[0111] S411: For each found entry, update its status indicating the entry's validity from pending to submitted.

[0112] S412: For the primary key of the same operated data, find the overlay state entry with the highest version number after the data operation among all overlay state entries with the status of committed, and take it as the final overlay state entry.

[0113] S413: Update the new state value after the data operation of the final superimposed state entry to the corresponding materialized view record in the materialized view storage, and complete the update;

[0114] S414: End.

[0115] As can be seen, this application can be applied to systems including medical application servers, hospital source databases storing patient diagnosis and treatment data, drug inventory data, clinic scheduling data, etc., medical message middleware, medical materialized view consumption services, and medical materialized view storage, such as hospital multi-system data synchronization platforms;

[0116] For example, in S1, a unique transaction identifier is extracted from the database transaction, such as the TxID assigned by the hospital source database to the patient diagnosis and treatment transaction: Tx_20240520_P001, where "P001" is the unique identifier prefix for the patient;

[0117] Extract the primary key and data operation type of the data being manipulated from the data manipulation instructions:

[0118] If the data operation instruction is to add a patient diagnosis record, the primary key of the data being operated on is the patient ID patient_id: P20240520001, and the data operation type is INSERT;

[0119] If the data operation instruction is to update the inventory of a certain type of medicine in the pharmacy, then the primary key of the data being operated on is the drug ID drug_id: D202405_A003, where "A003" is the drug category identifier, and the data operation type is UPDATE;

[0120] If the data operation command is to delete expired schedules in the consultation room, the primary key of the data being operated on is the consultation room ID consulting_room_id: R05_20240520, where "R05" is the consultation room number, and the data operation type is DELETE;

[0121] Obtain the generated information after the data operation command is executed. The generated information includes the new status value after the data operation, such as the diagnosis result "acute upper respiratory tract infection" in the patient's diagnosis record, the remaining quantity of medicine after the update "120 boxes", and the idle time period "2024-05-21 09:00-11:00" after the clinic schedule is deleted.

[0122] A distributed, unique, ordered ID generation algorithm, such as the snowflake algorithm, is used to generate version numbers for the same data after data operations. For example, for the diagnosis record of patient P20240520001, the version number of the operation at 10:00 is 1001, the version number of the supplementary diagnosis operation at 10:05 is 1002, and the version number of the subsequent diagnosis operation is higher, ensuring the time sequence accuracy of patient diagnosis data.

[0123] The unique transaction identifier Tx_20240520_P001, the primary key of the data being operated on (patient_id: P20240520001), the data operation type INSERT, the new status value after the data operation (diagnosis result "acute upper respiratory tract infection"), the attending physician ID "Dr0508", and the version number after the data operation (1001) are encapsulated together into a JSON format status intent event.

[0124] For clarification, "acquisition" in this application refers to obtaining the required content or data using existing technical means.

[0125] Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.

[0126] In the embodiments provided by this invention, it should be understood that the disclosed system or method can be implemented in other ways. For example, the embodiments of the invention described above are merely illustrative; for instance, the division of modules is only a logical functional division, and there may be other division methods in actual implementation.

[0127] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0128] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in the form of hardware plus software functional modules.

[0129] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the basic characteristics of the present invention.

[0130] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A data low-latency materialized view method applied to a system comprising an application server, a source database, a message middleware, a materialized view consumption service and a materialized view storage, characterized in that, The method comprises the following steps: S1, after the application server initiates a database transaction to a source database and executes a data operation instruction, a state intent event is constructed before the database transaction is committed; S2, the application server performs an asynchronous publishing operation of the state intent event, and sends the state intent event to a message middleware; S3, a memory superimposed state table is constructed in a materialized view consumption service, and the materialized view consumption service reads the state intent event from the message middleware; According to the state intent event, a superimposed state entry is generated in the memory superimposed state table; S4, after the database transaction is successfully committed, a commit state event is constructed in the source database, and the commit state event is transmitted to the materialized view consumption service; S5, after receiving the commit state event, the materialized view consumption service generates a final superimposed state entry in the memory superimposed state table, and updates a new state value after data operation in the final superimposed state entry to a corresponding materialized view record in the materialized view storage; In S1, the state intent event is constructed as follows: A unique transaction identifier is extracted from the database transaction; The primary key of the data to be operated and the data operation type are extracted from the data operation instruction; The generated information after the execution of the data operation instruction is obtained, and the generated information includes the new state value after the data operation; A distributed unique ordered ID generation algorithm is used to generate a version number after the data operation for the same data, and the version number after the data operation is used to identify the sequence of the data operation; The unique transaction identifier, the primary key of the data to be operated, the data operation type, the new state value after the data operation, and the version number after the data operation are collectively encapsulated as a state intent event in JSON format; In S3, the memory superimposed state table is constructed in the materialized view consumption service as follows: When the materialized view consumption service is started, a key-value pair data structure based on a primary key index is automatically initialized in the memory space of the materialized view consumption service as the basic framework of the memory superimposed state table; The primary key of the data to be operated is used as the unique index key, a hash index is used to generate a mapping rule between the primary key of the data to be operated and the superimposed state entry, and the memory superimposed state table is generated; In S4, the commit state event is constructed in the source database as follows: After the database transaction is successfully committed, the state of the database transaction is marked as a commit success state in the source database; The unique transaction identifier of the database transaction is extracted, and the unique transaction identifier of the database transaction and the commit success state are encapsulated in a preset format to form a commit state event.

2. The method of claim 1, wherein, In S3, according to the state intent event, a superimposed state entry is generated in the memory superimposed state table as follows: A corresponding superimposed state entry is generated for the primary key of the data to be operated in each state intent event; The superimposed state entry includes the primary key of the data to be operated, the unique transaction identifier, the new state value after the data operation, the version number after the data operation, and a state identifier for identifying the validity of the entry; The state identifier for identifying the validity of the entry is a pending state; When the primary key of the same data to be operated corresponds to multiple state intent events, and the primary key of the same data to be operated exists in the memory superposition state table, the materialized view consumption service compares the version number after the data operation of the newly received state intent event with the version number after the data operation recorded in the superposition state entry, and retains the superposition state entry corresponding to the state intent event with the highest version number after the data operation.

3. The method of claim 1, wherein, In S4, the commit state event is transmitted to the materialized view consumption service through a lightweight message channel, and the lightweight message channel is a pg_notify mechanism of a PostgreSQL database or a BINLOG EVENT notification mechanism of a MySQL database.

4. The method of claim 1, wherein, In S5, the materialized view consumption service generates a final superposition state entry in the memory superposition state table after receiving the commit state event, as follows: According to the transaction identifier in the commit state event, all superposition state entries containing the transaction identifier are found in the memory superposition state table; for each found superposition state entry, the state identifier of the superposition state entry is updated from the pending state to the committed state; For all superposition state entries of the same primary key of the data to be operated corresponding to the committed state, the superposition state entry with the highest version number after the data operation is selected as the final superposition state entry.

5. A data low-latency materialized view system for using the data low-latency materialized view method of any one of claims 1 to 4, characterized in that, Comprise: An application server, a source database, a message middleware, a materialized view consumption service and a materialized view storage, wherein: The application server is configured to initiate a database transaction to the source database and execute a data operation instruction, build a state intent event before the execution of the data operation instruction and before the commit of the database transaction, perform an asynchronous publishing operation on the state intent event, and send the state intent event to the message middleware; The message middleware is configured to receive and temporarily store the state intent event sent by the application server, and provide the materialized view consumption service with the state intent event; The source database is configured to mark the state of the database transaction as a commit success state after the commit of the database transaction, build a commit state event based on the unique transaction identifier of the database transaction and the commit success state, and transmit the commit state event to the materialized view consumption service through a lightweight message channel; The materialized view consumption service is configured to build a memory superposition state table, read the state intent event from the message middleware, generate a superposition state entry in the memory superposition state table based on the state intent event, receive the commit state event transmitted by the source database, update the state identifier of the identification entry of the superposition state entry according to the commit state event, and select a final superposition state entry; The materialized view storage is configured to receive the final superposition state entry sent by the materialized view consumption service, and update the new state value after the data operation in the final superposition state entry to the corresponding materialized view record; The state intent event is built as follows: Extract the unique transaction identifier from the database transaction; Extract the primary key of the data to be operated and the data operation type from the data operation instruction; Obtain the generated information after the execution of the data operation instruction, and the generated information includes the new state value after the data operation; A distributed unique ordered ID generation algorithm is used to generate a version number of the data after the data operation for the same data, and the version number of the data after the data operation is used to identify the sequence of the data operation; The unique transaction identifier, the primary key of the data to be operated, the data operation type, the new state value after the data operation, and the version number after the data operation are collectively encapsulated into a state intent event in a JSON format; An in-memory overlay state table is constructed in the materialized view consumption service, as follows: When the materialized view consumption service is started, a key-value pair data structure based on a primary key index is automatically initialized in the memory space of the materialized view consumption service as the infrastructure of the in-memory overlay state table; The primary key of the data to be operated is used as a unique index key, a hash index is used to generate a mapping rule of the primary key of the data to be operated and the overlay state entry, and the in-memory overlay state table is generated; A commit state event is constructed in the source database, as follows: After the database transaction is successfully committed, the state of the database transaction is marked as a commit success state in the source database; The unique transaction identifier of the database transaction is extracted, and the unique transaction identifier of the database transaction and the commit success state are encapsulated in a preset format to form a commit state event.

Images