A data storage method and device under a multi-active architecture in different places

Abstract

This invention discloses a data storage method and apparatus for a multi-site active-active architecture, relating to the field of computer technology. One specific implementation of the method includes: generating messages based on database operation logs, the messages including data identifiers; copying the messages according to the number of search engines, obtaining a number of messages equal to the number of search engines; parsing each message to obtain the data identifier corresponding to each message, and using a remote procedure call interface to query the latest data corresponding to the data identifier from the database; and storing the latest data in the corresponding search engine. This implementation can ensure the consistency of data storage under a multi-site active-active architecture, thereby effectively improving business continuity and data integrity. In the event of a disaster recovery failure, user traffic can be redirected to other normal search engines to ensure that businesses can access data normally, improving the stability and reliability of the business system.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data storage method and apparatus for a geographically distributed active-active architecture. Background Technology

[0002] With the expansion of information technology and the increase in business data, significant challenges have arisen in terms of database storage and query efficiency. Elasticsearch (ES), as a powerful, distributed, highly scalable, and real-time search and data analysis engine, easily enables the searching, analysis, and exploration of large amounts of data, supports real-time data storage and retrieval, and efficiently solves the problems of data storage and retrieval in business systems.

[0003] In the process of realizing this invention, the inventors discovered the following problems in the prior art:

[0004] In real-world applications, a geographically distributed, multi-active architecture is typically used to store data for search engines. While this effectively addresses system disaster recovery issues caused by operational errors, hardware failures, network attacks, network outages, power outages, and natural disasters, a geographically distributed, multi-active architecture requires ensuring data consistency during storage. Otherwise, business interruptions or even data loss can occur, impacting normal business activities and failing to meet the stability and reliability requirements of the business system. Therefore, ensuring data consistency under a geographically distributed, multi-active architecture is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a data storage method and apparatus under a multi-site active-active architecture. Messages are replicated according to the number of search engines, so that each search engine storing the data parses the same message. Data in the search engines is updated based on the latest data queried from the database via a remote call interface, thereby ensuring the consistency of data storage across various search engines under the multi-site active-active architecture. This effectively improves business continuity. In the event of a disaster recovery failure, user traffic can be redirected to other normal search engines to ensure that the business can access data normally, thus improving the stability and reliability of the business system.

[0006] To achieve the aforementioned objective, according to one aspect of the present invention, a data storage method under a geographically distributed active-active architecture is provided, comprising:

[0007] A message is generated based on the database operation log, and the message includes a data identifier;

[0008] The message is copied according to the number of search engines to obtain a number of messages equal to the number of search engines;

[0009] Each message is parsed to obtain the data identifier corresponding to each message, and the latest data corresponding to the data identifier is queried from the database using a remote call interface;

[0010] The latest data is stored in the corresponding search engine.

[0011] Optionally, consuming and parsing each message to obtain the data identifier corresponding to each message includes: iterating through the message list consisting of at least one message included in each message to obtain the set of data identifiers corresponding to each message.

[0012] Optionally, before parsing each message to obtain the data identifier corresponding to each message, the method further includes: generating a remote call interface corresponding to each message based on the remote call interface configuration information.

[0013] Optionally, before storing the latest data in the corresponding search engine, the method further includes: performing idempotency verification on the message based on the latest data and discarding duplicate messages.

[0014] Optionally, storing the latest data in a corresponding search engine includes: searching for corresponding data in the search engine based on the data identifier; if the data identifier does not exist in the search engine, inserting the data identifier and the latest data into the search engine; if the data identifier exists in the search engine, obtaining the data value corresponding to the data identifier, and updating the data value based on the latest data.

[0015] Optionally, the message further includes a message type, which includes deletion, update, and insertion; wherein, if the message type is deletion, the latest data corresponding to the data identifier retrieved from the database is empty; updating the data value according to the latest data includes: deleting the data identifier from the search engine.

[0016] According to a second aspect of the present invention, a data storage device under a geographically distributed active-active architecture is provided, comprising:

[0017] The message generation module is used to generate messages based on the database operation logs, and the messages include data identifiers;

[0018] The message copying module is used to copy the message according to the number of search engines, so as to obtain a number of messages equal to the number of search engines;

[0019] The data acquisition module is used to parse each message to obtain the data identifier corresponding to each message, and use a remote call interface to query the latest data corresponding to the data identifier from the database;

[0020] A storage module is used to store the latest data in the corresponding search engine.

[0021] Optionally, the data acquisition module is further configured to: iterate through the message list consisting of at least one message for each message to obtain a set of data identifiers corresponding to each message.

[0022] According to a third aspect of the present invention, a data storage electronic device under a geographically distributed active-active architecture is provided, characterized in that it comprises:

[0023] One or more processors;

[0024] Storage device for storing one or more programs.

[0025] When the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the embodiments of the present invention.

[0026] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0027] One embodiment of the invention has the following advantages or beneficial effects: By generating messages based on database operation logs, the messages include data identifiers; replicating the messages according to the number of search engines to obtain a number of messages equal to the number of search engines; parsing each message to obtain the corresponding data identifier, and using a remote call interface to query the database for the latest data corresponding to the data identifier; and storing the latest data in the corresponding search engine, the technical solution achieves message replication based on the number of search engines, enabling each search engine storing data to parse the same message and update the data in the search engine based on the latest data queried from the database via the remote call interface. This ensures the consistency of data storage in each search engine under a multi-site active-active architecture, thereby effectively improving business continuity. In the event of a disaster recovery failure, user traffic can be redirected to other normal search engines to ensure that the business can access data normally, improving the stability and reliability of the business system. Attached Figure Description

[0028] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0029] Figure 1 This is a schematic diagram of the main flow of a data storage method under a multi-site active-active architecture according to an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of the overall design scheme of ES dual-cluster dual-write consistency according to an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of the main process of ES dual-cluster dual-write consistency in an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of the main message processing flow for ES dual-cluster dual-write consistency in an embodiment of the present invention;

[0033] Figure 5 This is a schematic diagram of the main modules of a data storage device under a multi-site active-active architecture according to an embodiment of the present invention;

[0034] Figure 6 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0035] Figure 7 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0036] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0037] In real-world applications, a geographically distributed, multi-active architecture is typically used to store data for search engines. While this effectively solves system disaster recovery issues caused by operational errors, hardware failures, network attacks, network outages, power outages, and natural disasters, data storage inconsistency issues can lead to business interruptions or even data loss, thereby affecting normal business activities and failing to meet the stability and reliability requirements of business systems.

[0038] To address the aforementioned problems in existing technologies, this invention proposes a data storage method for a multi-site active-active architecture. Messages are replicated based on the number of search engines, ensuring that each search engine parses the same message and updates its data based on the latest data queried from the database via a remote API. This guarantees data consistency across search engines within the multi-site active-active architecture, effectively improving business continuity. In the event of a disaster recovery issue, user traffic can be redirected to other working search engines to ensure uninterrupted data access for businesses, thus enhancing the stability and reliability of the business system.

[0039] In the description of the embodiments of the present invention, the terms involved and their meanings are as follows:

[0040] ES: Elasticsearch, a search server based on Lucene. It provides a distributed, multi-user-capable full-text search engine and is a popular enterprise-grade search engine;

[0041] Disaster recovery refers to establishing two or more functionally identical IT systems in geographically dispersed locations. These systems can monitor each other's health and switch functions. If one system stops working due to an accident (such as a fire or earthquake), the entire application system can switch to the other, allowing the system to continue functioning normally. Disaster recovery technology is a component of high availability technology. Disaster recovery systems place greater emphasis on handling the impact of external environmental factors on the system, especially the impact of catastrophic events on the entire IT node, providing node-level system recovery capabilities.

[0042] Idempotency: For the same system, under the same conditions, if a single request and repeated requests have the same impact on the resource, then the operation is said to be idempotent.

[0043] JMQ: A message middleware platform that provides reliable message and data delivery with high availability, scalability, and maintainability;

[0044] RPC: Remote Procedure Call;

[0045] In computer authentication, "token" means a temporary token; in lexical analysis, it means a mark or token. It is generally used for invitations and logins to systems.

[0046] binary log: used to record all changes and modifications in the database;

[0047] binlake: binlake is a clustered database binary log management, collection, and distribution system, and it transparently integrates with message distribution and subscription systems such as JMQ and Kafka.

[0048] Figure 1 This is a schematic diagram illustrating the main flow of a data storage method under a geographically distributed active-active architecture according to an embodiment of the present invention, as shown below. Figure 1 As shown, the data storage method under the multi-site active-active architecture of this invention includes the following steps S101 to S104.

[0049] Step S101: Generate a message based on the database operation log, the message including a data identifier.

[0050] Specifically, as business volume increases, traditional databases can no longer adequately meet actual needs in terms of execution efficiency. By storing data on a search engine, efficient storage and retrieval of business data can be achieved. Synchronizing data with the search engine can be accomplished using the database's binary log. In practical applications, to improve development efficiency and shorten development cycles, binlake is used to automatically collect, distribute, and subscribe to the database's binary log. Binlake encapsulates database operations, corresponding data, and data identifiers into messages for message queue processing.

[0051] Step S102: Copy the message according to the number of search engines to obtain a message in the same number of copies as the number of search engines.

[0052] Specifically, to avoid business interruptions and data loss caused by the aforementioned disaster recovery issues in the business system, and to improve system high availability, a multi-site active-active architecture is used to implement the multi-cluster design of the search engine. Multiple search engines are distributed across different geographical locations. The number of message consumption groups is determined based on the number of search engines, and multiple message consumption groups are established, each corresponding to a search engine in a specific geographical location. Messages in the message queue are replicated according to the message consumption groups, resulting in a message count equal to the number of search engines, thus achieving data synchronization between the geographically distributed search engines.

[0053] Step S103: Parse each message to obtain the data identifier corresponding to each message, and use a remote call interface to query the latest data corresponding to the data identifier from the database.

[0054] According to one embodiment of the present invention, before parsing each message to obtain the data identifier corresponding to each message, the method further includes: generating a remote call interface corresponding to each message based on remote call interface configuration information.

[0055] Specifically, considering the risk of message backlog caused by binlake, which could lead to inaccurate data synchronized to the search engine, a Remote Procedure Call (RPC) interface is used to retrieve the latest data from the database to improve data accuracy. Before parsing each message to obtain its corresponding data identifier, the number of message consumption groups and the fully qualified name, alias, and token of the RPC interface are configured. This dynamic generation of message consumption groups and RPC interfaces is achieved using relevant configuration information from the search engine's Elasticsearch (ES) configuration table.

[0056] For example, the main configuration information of Elasticsearch includes: a unique ID for identifying the business application and its name; a topic for allocating message storage corresponding to the business; the number of groups for dynamically generating message consumption groups; a group prefix for distinguishing different business lines; the fully qualified name, alias, and token for dynamically generating RPC interfaces; and configuration information for configuring the basic information of the search engine itself, such as the Elasticsearch index, Elasticsearch type, and number of brokers.

[0057] By establishing the ES configuration information described above, RPC call interfaces can be dynamically generated and multiple message consumer groups can be dynamically configured, providing the necessary conditions for the implementation of a multi-site active-active architecture.

[0058] According to another embodiment of the present invention, consuming and parsing each message to obtain the data identifier corresponding to each message includes: circumventing the message list consisting of at least one message included in each message to obtain the set of data identifiers corresponding to each message.

[0059] Specifically, for each message consumption group, by listening to the topic messages configured above, the binlake messages generated by the operation logs are obtained, and the binlake messages are parsed to obtain the operations, data, and data identifiers included in the messages. For cases where the message is a message list consisting of more than one message, such as batch modification operations, the message list needs to be parsed by looping through it to obtain the set of data identifiers corresponding to the messages.

[0060] Based on the data identifier obtained above, and using the data identifier as the primary key, the latest data corresponding to the data identifier is queried from the database using the RPC interface to obtain accurate message data, thus solving the problem of inaccurate data caused by message backlog.

[0061] Step S104: Store the latest data in the corresponding search engine.

[0062] According to one embodiment of the present invention, before storing the latest data in the corresponding search engine, the method further includes: performing idempotency verification on the message based on the latest data, and discarding duplicate messages.

[0063] Specifically, in actual business requests, due to system failures, request timeouts, and other reasons, the same request may be initiated multiple times. This requires updating the same record multiple times, which not only puts a significantly increased operational burden on the search engine, but also has more serious consequences for transactions involving payments or those that only allow one-time operations. To solve this problem, it is necessary to perform idempotency verification on messages and discard duplicate submitted message data, which alleviates the operational pressure on the search engine while maintaining the consistency of data storage.

[0064] According to one embodiment of the present invention, storing the latest data in a corresponding search engine includes: searching for corresponding data in the search engine based on the data identifier; if the data identifier does not exist in the search engine, inserting the data identifier and the latest data into the search engine; if the data identifier exists in the search engine, obtaining the data value corresponding to the data identifier, and updating the data value based on the latest data.

[0065] According to another embodiment of the present invention, the message further includes a message type, which includes deletion, update, and insertion; wherein, if the message type is deletion, the latest data corresponding to the data identifier retrieved from the database is empty; updating the data value according to the latest data includes: deleting the data identifier from the search engine.

[0066] Specifically, the messages generated from the database operation logs include message types, which include operations such as deletion, update, and insertion of message data. If the message type is update, the corresponding data is retrieved from the search engine using the data identifier, and the data in the search engine is updated according to the latest data retrieved from the database. If the message type is insertion, the data identifier and the corresponding latest data are inserted into the search engine. If the message type is deletion, the latest data corresponding to the data identifier retrieved from the database is empty, and the data identifier needs to be deleted from the search engine.

[0067] The following section will use consistent data storage on two Elasticsearch clusters as an example to illustrate the specific implementation process of this invention.

[0068] Figure 2 This is a schematic diagram of the overall design scheme of ES dual-cluster dual-write consistency according to an embodiment of the present invention. In the diagram, each business line (insurance business, tasks, wish list, claims settlement and system logs, etc.) realizes database storage and management of business data through corresponding applications (insurance business application, task application, wish list application, claims settlement application and system log application, etc.). Binlake, as an automated collection, distribution and subscription system for binary logs, listens to and collects data in the database and generates corresponding messages, which are sent to MQ messages. The messages are copied and parsed according to the number of ES clusters to obtain the data identifier in the message. The latest data corresponding to the data identifier is obtained through the RPC interface. Finally, the latest data is stored in the ES data centers in locations A and B, realizing ES dual-cluster dual-write.

[0069] Figure 3This is a schematic diagram of the main process of ES dual-cluster dual-write consistency processing in an embodiment of the present invention. Based on the configuration table above, an RPC interface is dynamically generated, and message consumption groups (e.g., consumer group 1 and consumer group 2) are established, where each message consumption group corresponds to one ES cluster. Binlake is started upon receiving the ACK confirmation information; in response to application requests for database insertion, modification, and deletion operations, messages are generated via binlake; based on the number of ES clusters, the binlake message is copied to obtain two binlake messages, which are stored in the two message consumption groups (consumer group 1 and consumer group 2) respectively; based on the parsing of the binlake message, the latest data is obtained using the RPC interface; after idempotency verification, duplicate messages are discarded, and the data is stored in ES data center 1 at location A and ES data center 2 at location B.

[0070] Figure 4 This is a schematic diagram of the main message processing flow for dual-cluster dual-write consistency in Elasticsearch (ES) according to an embodiment of the present invention. The process involves: allocating topics corresponding to the business based on the ES configuration table; dynamically generating message consumer group 1 and message consumer group 2 with prefixes based on the number of consumer groups and prefixes in the ES configuration table; dynamically generating RPC interfaces based on the fully qualified name, alias, and token of the RPC interface; listening to messages on the configured topics and replicating messages according to the number of ES clusters to obtain two sets of binlake messages; obtaining the message list for each set of binlake messages; parsing the message list to obtain the message type (delete, update, and insert) and data identifier; querying the database for the latest data corresponding to the data identifier using the RPC interface; and synchronously storing the data in ES according to the data type (delete, update, and insert) after idempotency verification.

[0071] Correspondingly, for a multi-cluster, multi-write architecture with multiple active-active nodes, a similar approach can be adopted: obtain messages based on the database operation logs; replicate the obtained messages according to the number of Elasticsearch nodes (ES), which could be three messages from three ES nodes or five messages from five ES nodes; parse each message to obtain the data type and data identifier; and use the RPC interface to obtain the latest data corresponding to the data identifier from the database, so as to perform the latest data storage operation according to the data type.

[0072] Through the above embodiments, based on the synchronous data messages generated from the operation logs, the data messages are replicated according to the number of Elasticsearch instances (ES), and the latest data is obtained by querying the database using an RPC interface. Combined with the verification of data idempotency, the consistency of data storage under the multi-site active-active architecture of the present invention is guaranteed.

[0073] Figure 5 This is a schematic diagram of the main modules of a data storage device under a geographically distributed active-active architecture according to an embodiment of the present invention. Figure 5 As shown, the data storage device 500 under the multi-site active-active architecture mainly includes a message generation module 501, a message replication module 502, a data acquisition module 503, and a storage module 504.

[0074] Message generation module 501 is used to generate messages based on the database operation logs, the messages including data identifiers;

[0075] The message copying module 502 is used to copy the message according to the number of search engines, so as to obtain a number of messages equal to the number of search engines;

[0076] The data acquisition module 503 is used to parse each message to obtain the data identifier corresponding to each message, and use a remote call interface to query the latest data corresponding to the data identifier from the database;

[0077] Storage module 504 is used to store the latest data in the corresponding search engine.

[0078] Specifically, the data acquisition module 503 can also be used to: iterate through the message list consisting of at least one message for each message to obtain the data identifier set corresponding to each message.

[0079] Specifically, the data storage device 500 further includes an interface generation module (not shown in the figure), which is used to generate a remote call interface corresponding to each message based on the remote call interface configuration information before parsing each message to obtain the data identifier corresponding to each message.

[0080] Specifically, the data storage device 500 further includes an idempotency verification module (not shown in the figure), used to: perform idempotency verification on the message based on the latest data before storing the latest data in the corresponding search engine, and discard duplicate messages.

[0081] Specifically, the storage module 504 can also be used to: search for corresponding data in a search engine based on the data identifier; if the data identifier does not exist in the search engine, insert the data identifier and the latest data into the search engine; if the data identifier exists in the search engine, obtain the data value corresponding to the data identifier, and update the data value based on the latest data.

[0082] Specifically, the message also includes message types, which include deletion, update, and insertion; wherein, if the message type is deletion, the latest data corresponding to the data identifier retrieved from the database is empty; the storage module 504 can also be used to delete the data identifier from the search engine.

[0083] Figure 6 This is an exemplary system architecture diagram 600 in which embodiments of the present invention can be applied.

[0084] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, and 603, a network 604, and a server 605. Network 604 serves as the medium for providing communication links between terminal devices 601, 602, and 603 and server 605. Network 604 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0085] Users can use terminal devices 601, 602, and 603 to interact with server 605 via network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as data storage applications in a multi-site active-active architecture (for example only).

[0086] Terminal devices 601, 602, and 603 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0087] Server 605 can be a server providing various services, such as a backend management server supporting data storage in a multi-site active-active architecture using terminal devices 601, 602, and 603 (this is just an example). The backend management server can generate messages based on database operation logs, the messages including data identifiers; copy the messages according to the number of search engines, obtaining a number of messages equal to the number of search engines; parse each message to obtain the corresponding data identifier, and use a remote call interface to query the database for the latest data corresponding to the data identifier; store the latest data in the corresponding search engine for processing, and feed back the processing results (e.g., stored data – this is just an example) to the terminal devices.

[0088] It should be noted that the data storage method under the geographically distributed active-active architecture provided in the embodiments of the present invention is generally executed by the server 605, and correspondingly, the data storage device under the geographically distributed active-active architecture is generally set in the server 605.

[0089] It should be understood that Figure 6 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0090] The following is for reference. Figure 7It shows a schematic diagram of the structure of a computer system 700 suitable for implementing terminal devices or servers of the present invention. Figure 7 The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0091] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the system 700. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0092] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0093] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs the functions defined above in the system of this invention.

[0094] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0095] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0096] The units described in the embodiments of the present invention can be implemented in software or in hardware. The described units can also be housed in a processor; for example, a processor can be described as including: a message generation module, a message copying module, a data acquisition module, and a storage module.

[0097] In some cases, the names of these modules do not constitute a limitation on the module itself. For example, a message generation module can also be described as "a module for generating messages based on the operation logs of a database, wherein the messages include data identifiers".

[0098] On the other hand, the present invention also provides a computer-readable medium, which may be included in the device described in the embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to: generate messages based on database operation logs, the messages including data identifiers; copy the messages according to the number of search engines, obtaining a number of messages equal to the number of search engines; parse each message to obtain a data identifier corresponding to each message, and query the database for the latest data corresponding to the data identifier using a remote procedure call interface; and store the latest data in the corresponding search engine.

[0099] The technical solution of the present invention has the following advantages or beneficial effects: By generating messages based on the database operation logs, the messages include data identifiers; replicating the messages according to the number of search engines to obtain a number of messages equal to the number of search engines; parsing each message to obtain the data identifier corresponding to each message, and using a remote call interface to query the latest data corresponding to the data identifier from the database; and storing the latest data in the corresponding search engine, the technical solution achieves message replication based on the number of search engines, enabling each search engine to parse the same message and update the data in the search engine based on the latest data queried from the database via the remote call interface. This ensures the consistency of data storage in each search engine under a multi-site active-active architecture, thereby effectively improving business continuity. In the event of a disaster recovery failure, user traffic can be redirected to other normal search engines to ensure that the business can access data normally, thus improving the stability and reliability of the business system.

[0100] The specific embodiments described herein do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data storage method under a geographically dispersed multi-active architecture, characterized in that, The method comprises the following steps: generating a message according to an operation log of a database by using a binlake system, wherein the message comprises a data identifier; duplicating the message according to a number of search engines to obtain a number of messages equal to the number of search engines; duplicating the message according to the number of search engines, comprising: distributing a plurality of search engines in different geographical locations, determining a number of message consumption groups according to the number of search engines, establishing a plurality of message consumption groups, and each message consumption group corresponding to a search engine in a geographical location; and duplicating the message according to the message consumption group; parsing each message to obtain a data identifier corresponding to each message, and querying the latest data corresponding to the data identifier from the database by using a remote call interface; storing the latest data into the corresponding search engine.

2. The method of claim 1, wherein, The parsing of each message to obtain the data identifier corresponding to each message comprises: cycling through a message list comprising no less than one message included in each message to obtain a data identifier set corresponding to each message.

3. The method of claim 1, wherein, Before the parsing of each message to obtain the data identifier corresponding to each message, the method further comprises: generating a remote call interface corresponding to each message according to remote call interface configuration information.

4. The method of claim 1, wherein, Before the storing of the latest data into the corresponding search engine, the method further comprises: verifying the idempotency of the message according to the latest data, and discarding a duplicate message.

5. The method of claim 1, wherein, The storing of the latest data into the corresponding search engine comprises: finding corresponding data from the search engine according to the data identifier; in a case where the data identifier does not exist in the search engine, inserting the data identifier and the latest data into the search engine; in a case where the data identifier exists in the search engine, obtaining a data value corresponding to the data identifier, and updating the data value according to the latest data.

6. The method according to claim 1 or 5, characterized in that, The message further comprises a message type, and the message type comprises deletion, update and insertion; wherein, if the message type is deletion, the latest data corresponding to the data identifier queried from the database is empty; updating the data value according to the latest data comprises: deleting the data identifier from the search engine.

7. A data storage apparatus under a geo-multiplication active architecture, characterized in that, The method comprises the following steps: a message generation module for generating a message according to an operation log of a database by using a binlake system, wherein the message comprises a data identifier; a message duplication module for duplicating the message according to a number of search engines to obtain a number of messages equal to the number of search engines; the message duplication module is further configured to distribute a plurality of search engines in different geographical locations, determine a number of message consumption groups according to the number of search engines, establish a plurality of message consumption groups, and each message consumption group corresponding to a search engine in a geographical location; and duplicate the message according to the message consumption group; a data acquisition module for parsing each message to obtain a data identifier corresponding to each message, and querying the latest data corresponding to the data identifier from the database by using a remote call interface; A storage module is configured to store the latest data into a corresponding search engine.

8. The apparatus of claim 7, wherein, The data obtaining module is further configured to: for each message, obtaining a message list including no less than one message included in the message, and traversing the message list to obtain a data identifier set corresponding to the each message.

9. A mobile electronic device terminal, characterized by comprising: one or more processors; a memory device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method according to any one of claims 1-6.

10. A computer readable medium having stored thereon a computer program, characterized in that, The program, when executed by a processor, implements the method according to any one of claims 1-6.

Images