User Operation Request Information Synchronization and Batch Persistence Processing System

Abstract

Embodiments of the present disclosure disclose a system for synchronizing user operation request information and batch persistent processing. A specific implementation manner of the system includes: business node clusters in each region, off-site persistent nodes clusters, and an information persistent database. Among them, the off-site persistent nodes clusters include each off-site persistent node. For each business node cluster in the business node clusters in each region, the business node cluster in the region is configured as follows: the region business master node included in the business node cluster in the region is configured to perform the following: generate a request and return message log file; the region business standby node is configured to perform the following synchronization step: send the request and return message log file at the top of the preset stack to the off-site persistent node; the off-site persistent node is configured to perform the following batch persistent processing: perform batch database storage processing on each database storage operation statement information. This implementation manner reduces network overhead and the occupation of hardware resources.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technologies, and more particularly, to a system for synchronizing user operation request information and batch persistent processing. Background Art

[0002] As business systems (such as trading systems and user management systems) present a distributed and multi-node architecture, data is scattered in different regions. A system for synchronizing user operation request information and batch persistent processing is a system that synchronizes and persists user operation request information and operation return information scattered in different regions to a central database. Currently, when synchronizing and persisting user operation request information scattered in different regions to a central database, the commonly adopted method is to deploy an independent database in each region and synchronize the user operation request information and operation return information within the region to the central database through the built-in synchronization function of the database or a third-party tool.

[0003] However, when adopting the above method to synchronize user operation request information and operation return information scattered in different regions to a central database, the following technical problems often exist:

[0004] By deploying an independent database in each region and synchronizing the user operation request information and operation return information within the region to the central database through the built-in synchronization function of the database and a third-party tool, database synchronization data usually requires frequent network interactions and transaction submissions, resulting in a large network overhead and a high data synchronization delay. At the same time, by deploying an independent database in each region, the database instance needs to occupy an independent server and storage device, resulting in a high occupancy of hardware resources.

[0005] The above information disclosed in this background art section is only used to enhance the understanding of the background of the inventive concept, and thus, it may include information that does not form the prior art known to those of ordinary skill in the art. Summary of the Invention

[0006] The content part of the present disclosure is used to briefly introduce concepts that will be described in detail in the subsequent detailed implementation part. The content part of the present disclosure is not intended to identify the key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0007] Some embodiments of the present disclosure propose a system for synchronizing user operation request information and batch persistent processing and a method for batch persistent processing of user operation request and return information to solve one or more of the technical problems mentioned in the above background art section.

[0008] In a first aspect, some embodiments of the present disclosure provide a user operation request information synchronization and batch persistent processing system, which includes: business node clusters in each region, off-site persistent nodes clusters, and an information persistent database. Among them, the off-site persistent nodes clusters include each off-site persistent node, and each off-site persistent node in the above-mentioned off-site persistent nodes corresponds to one business node cluster in the above-mentioned business node clusters in each region. The business node clusters in each region include a primary business node and a standby business node in the region; for each business node cluster in the above-mentioned business node clusters in each region, the following configurations are performed on the business node cluster in the region: the primary business node included in the business node cluster in the region is configured to perform the following replication steps: receiving user operation requests and return information within a preset time, and generating a request and return message log file corresponding to the user operation requests and return information; copying the request and return message log file to the standby business node included in the business node cluster in the region; the standby business node in the region is configured to perform the following synchronization steps: storing the request and return message log file copied from the primary business node in the region to a preset stack; monitoring the log reporting status of the primary business node in the region to obtain log reporting status monitoring information; in response to determining that the log reporting status monitoring information indicates that the primary node log reporting service is interrupted, sending the request and return message log file at the top of the preset stack to the off-site persistent node corresponding to the business node cluster in the region in the off-site persistent nodes cluster; the off-site persistent node is configured to perform the following batch persistent processing: generating respective database operation statement information based on the request and return message log file sent by the standby business node in the region; performing batch database operations on the respective database operation statement information to update the information persistent database.

[0009] In a second aspect, some embodiments of the present disclosure provide a method for generating building knowledge reply information, which includes: receiving a request and return message log file sent by a standby business node in a region, where the request and return message log file contains respective message data representing user operation requests and return information; generating respective database operation statement information based on the request and return message log file; performing batch database operations on the respective database operation statement information to update the information persistent database.

[0010] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: Through the user operation request information synchronization and batch persistence processing system of some embodiments of the present disclosure, network overhead, data synchronization delay, and the occupation of hardware resources are reduced. Specifically, the reasons for the large network overhead, data synchronization delay, and high occupation of hardware resources are as follows: Independent databases are deployed in each region, and the user operation request information and operation return information within the region are synchronized to the central database through the built-in synchronization function of the database and third-party tools. Database synchronization usually requires frequent network interactions and transaction submissions, resulting in a large network overhead and a high data synchronization delay. At the same time, independent databases are deployed in each region, and database instances need to occupy independent servers and storage devices, resulting in a high occupation of hardware resources. Based on this, the user operation request information synchronization and batch persistence processing system of some embodiments of the present disclosure includes: business node clusters in each region, off-site persistence node clusters, and information persistence databases. Among them, the above-mentioned off-site persistence node clusters include each off-site persistence node, and each off-site persistence node in the above-mentioned off-site persistence nodes corresponds to a business node cluster in one of the above-mentioned business node clusters in each region. The above-mentioned business node clusters in each region include a business main node and a business standby node in the region. For each business node cluster in the above-mentioned business node clusters in each region, the following configurations are performed on the business node cluster in the region: First, the business main node included in the business node cluster in the region is configured to perform the following replication steps: The first step is to receive user operation requests and return information within a preset time, and generate a request and return message log file corresponding to the above user operation requests and return information. Thus, a request and return message log file corresponding to the above user operation requests and return information can be generated. The second step is to copy the above request and return message log file to the business standby node included in the business node cluster in the region. Thus, the request and return message log file can be copied to the business standby node. Thus, the user operation requests and return information can be collected by the business main node, and the request and return message log file can be copied to the business standby node. The above-mentioned business standby node is configured to perform the following synchronization steps: First, store the request and return message log file copied from the business main node to a preset stack. Thus, the request and return message log file can be stored in the preset stack so that when the main node log reporting service is interrupted later, the request and return message log file at the top of the stack can be retrieved in time. Then, monitor the log reporting status of the above business main node to obtain log reporting status monitoring information. Thus, the log reporting status of the business main node can be monitored.After determining that the log reporting status monitoring information indicates an interruption in the main node log reporting service, the request and response message log file at the top of the above-mentioned preset stack is sent to the off-site persistent node in the off-site persistent node cluster corresponding to the above-mentioned regional service node cluster. Thus, when the main node log reporting service is interrupted, the request and response message log file at the top of the stack (i.e., the latest one) can be synchronized to the off-site persistent node in the off-site persistent node cluster corresponding to the above-mentioned regional service node cluster. Thus, through the above-mentioned regional service standby node, when the main node log reporting service is interrupted, the request and response message log file at the top of the stack (i.e., the latest one) can be synchronized to the off-site persistent node. The above-mentioned off-site persistent node is configured to perform the following batch persistence processing: First, based on the request and response message log file sent by the above-mentioned regional service standby node, generate each statement information for the database storage operation. Thus, each statement information for batch database storage processing of the user operation requests and response information in the request and response message log file can be generated. Then, perform batch database storage processing on the above-mentioned each statement information for database storage operation to update the above-mentioned information persistent database. Thus, through the off-site persistent node, the information stored in the request and response message log file can be synchronized and persisted to the information persistent database (i.e., the central database). Also, because only the request and response message log file is transmitted during the process of synchronizing and persisting the received user operation requests and response information to the information persistent database (i.e., the central database), there is no need for frequent network interaction and transaction submission, reducing network overhead and synchronization latency. At the same time, through the user operation request information synchronization and batch persistence processing system, the data user operation request information and operation response information collected by the regional service main node are synchronized to the information persistent database, without the need to deploy a complete database in each region, only need to record logs and send messages, and the hardware resource occupancy is less. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In conjunction with the accompanying drawings and with reference to the following detailed description, the above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic and the elements and elements are not necessarily drawn to scale.

[0012] Figure 1 is an architectural diagram of an exemplary system of a user operation request information synchronization and batch persistence processing system according to the present disclosure;

[0013] Figure 2 Flowchart of some embodiments of a method for batch persistence processing of user operation requests and response information according to the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0014] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0015] In addition, it should be noted that for the sake of convenience of description, only parts related to the relevant invention are shown in the drawings. Without conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other.

[0016] It should be noted that the concepts such as "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence relationship of the functions performed by these devices, modules or units.

[0017] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly specified in the context, it should be understood as "one or more".

[0018] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0019] The present disclosure will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0020] Figure 1 An exemplary system architecture 100 of a heterogeneous data exchange system to which some embodiments of the present disclosure can be applied is shown.

[0021] As Figure 1As shown, the system architecture 100 may include: clusters of regional business nodes 101, clusters of off-site persistence nodes 102, and an information persistence database 103. Among them, the above-mentioned clusters of off-site persistence nodes 102 include each off-site persistence node 104, and each off-site persistence node in the above-mentioned each off-site persistence node corresponds to one cluster of regional business nodes in the above-mentioned clusters of regional business nodes 101. The above-mentioned clusters of regional business nodes include a regional business master node and a regional business standby node. The above-mentioned regional business master node and the above-mentioned regional business standby node can be connected through the medium of a communication link. The above-mentioned regional business standby node and the corresponding off-site persistence node in the above-mentioned clusters of off-site persistence nodes 102 can be connected through the medium of a communication link. The medium of the above-mentioned communication link can be a physical medium or method for transmitting data, such as a wired or wireless communication link or an optical fiber cable, etc. Among them, the above-mentioned information persistence database 103 can be a database for storing business data collected by each cluster of regional business nodes. The above-mentioned business data may include user operation requests and return information (such as customer requests, exchange return data, etc.).

[0022] It should be noted that each node in this application can be hardware, software, or even a function in software.

[0023] In some embodiments, for each cluster of regional business nodes 101 among the above-mentioned clusters of regional business nodes, the following configurations are performed on the above-mentioned clusters of regional business nodes:

[0024] The regional business master node included in the above-mentioned clusters of regional business nodes can be configured to perform the following replication steps:

[0025] Receive user operation requests and return messages within a preset time, and generate request and return message log files corresponding to the above user operation requests and return messages. Among them, the above user operation requests and return messages include each user operation request message and each operation return message. Among them, each user operation request message in the above user operation request messages can represent the request data sent by the user when performing an operation. The above user operation request information may include operation type information, operation object information, operating user information, and related parameter information. The above operation object information can represent the object for value operation. The above operating user information can be a user identifier. Each operation return message in the above operation return messages can be the operation data of the system after receiving the user operation request and performing the corresponding operation. For example, the above user operation request information can be "operation type information: value inflow operation (for example, buy), operation object information: 000001 (for example, virtual item (such as stock) or physical item (such as pencil) labeled 000001), operating user information: 123456789, and related parameter information: operation quantity: 100". For example, the above operation return message can be "operation type information: operation completed (for example, transaction completed), operation object information: 000001 (for example, virtual item (such as stock) or physical item (such as pencil) labeled 000001), operating user information: 123456789, and related parameter information: operation quantity: 100".

[0026] Copy the above request and return message log file to the regional business backup node included in the above regional business node cluster.

[0027] In some optional implementation manners of some embodiments, the above regional business master node can receive user operation requests and return messages within a preset time, and generate request and return message log files corresponding to the above user operation requests and return messages through the following steps:

[0028] First step, receive each user operation request message and each operation return message as user operation requests and return messages from a preset data source within a preset time period. Among them, the above preset data source can be a preset system (such as a preset trading system, a preset counter system, etc.) that sends each user operation request message and each operation return message to the regional business master node.

[0029] In the second step, convert the above user operation requests and return messages into respective message data, where each piece of message data among the above-mentioned message data corresponds to one piece of information among the above-mentioned user operation request messages and the above-mentioned operation return messages. In practice, the above regional service master node can call a preset data format processing library (for example, a JSON library, an XML library) to convert each user operation request message and each operation return message included in the above user operation requests and return messages into respective message data. Optionally, the above regional service master node can convert each user operation request message and each operation return message included in the above user operation requests and return messages into respective message data through a preset message generation tool (for example, a message generator).

[0030] In the third step, store the above respective message data in a preset log file.

[0031] In the fourth step, determine the preset log file storing the above respective message data as the request and return message log file corresponding to the above user operation requests and return messages.

[0032] In some optional implementation manners of some embodiments, the above regional service master node can copy the above request and return message log file to the regional service standby node included in the above regional service node cluster through the following steps:

[0033] Synchronize the above request and return message log file to the regional service standby node included in the above regional service node cluster through a preconfigured state machine replication mechanism. In practice, the above regional service master node can synchronize the request and return message log file to the regional service standby node included in the above regional service node cluster through a preconfigured consensus mechanism based on state machine replication (SMR) (such as the Paxos, Raft consensus mechanisms).

[0034] In some embodiments, the above regional service standby node can be configured to perform the following synchronization steps:

[0035] First, store the request and return message log file copied from the above regional service master node in a preset stack. Among them, the above preset stack can be a last-in, first-out stack for storing the request and return message log files transmitted by the regional service master node. It should be noted that the request and return message log file at the top of the preset stack can be the most recently transmitted request and return message log file.

[0036] Second, monitor the log reporting status of the above regional service master nodes to obtain log reporting status monitoring information. Among them, the above log reporting status monitoring information can be represented by a boolean value or a status code. As an example, the above log reporting status monitoring information can be "true" to indicate that the log reporting service of the master node is interrupted. Or the above log reporting status monitoring information can be "false" to indicate that the log reporting service of the master node is not interrupted.

[0037] Third, in response to determining that the log reporting status monitoring information indicates that the log reporting service of the master node is interrupted, send the request and response message log file at the top of the above preset stack to the off-site persistent node corresponding to the above regional service node cluster in the above off-site persistent node cluster.

[0038] In some optional implementation manners of some embodiments, the above regional service standby node can monitor the log reporting status of the above regional service master node through the following steps to obtain log reporting status monitoring information:

[0039] First step, receive the log reporting status information sent by the above regional service master node.

[0040] Second step, in response to determining that the above log reporting status information indicates that the log reporting fails, determine the information indicating that the log reporting service of the master node is interrupted as the log reporting status monitoring information.

[0041] In some embodiments, the above off-site persistent node can be configured to perform the following batch persistent processing:

[0042] First, generate each database insertion operation statement information based on the request and response message log file sent by the above regional service standby node.

[0043] Second, perform batch database insertion processing on the above each database insertion operation statement information to update the above information persistent database.

[0044] In some optional implementation manners of some embodiments, the above off-site persistent node can generate each database insertion operation statement information based on the request and response message log file sent by the above regional service standby node through the following steps:

[0045] First step: Determine each message data included in the above request and return message log file as each message data to be converted. Each message data to be converted among the above message data to be converted includes operation type information, primary key value information, and field information. The above operation type information can represent the operation type. For example, value inflow operations (such as buying), value outflow operations (such as selling), operation completion (such as transaction), etc. The above primary key value information can represent operation object information. The above field information can represent operating user information and relevant parameter information corresponding to the user operation. The above message data to be converted can be a message representing user operation request information or operation return information.

[0046] Second step: For each message data to be converted among the above message data to be converted, perform the following conversion steps:

[0047] First sub-step: Determine the operation type information included in the above message data to be converted as the operation type information to be queried.

[0048] Second sub-step: Query the target database operation statement template information corresponding to the above operation type information to be queried from the preset mapping relationship table of operation statements. The above preset mapping relationship table of operation statements can be a database table or configuration file preset for establishing the mapping relationship between operation type information and database operation statement template information. The above database operation statement template information can be a standardized template for describing database operations (such as insertion, update, deletion, etc.). As an example, the database operation statement template representing insertion can be:

[0049] “INSERT INTO {table_transactions} (object_id, {fields}) VALUES({object_id _value}, {field_values})”

[0050] The above {table_transactions} can represent the target table name (such as transactions). The above {object_id _value} can be a placeholder representing the primary key value information for storing operation object information. The above {fields} can be a placeholder representing the field information for storing relevant parameter information of the user operation.

[0051] The third sub-step is to insert the primary key value information and field information included in the above-mentioned message data to be converted into the database operation statement template corresponding to the above-mentioned target database operation statement template information, so as to update the above-mentioned database operation statement template. In practice, the above-mentioned off-site persistence node can insert the primary key value information into the position where the placeholder representing the primary key value information is located in the above-mentioned database operation statement template, and can insert the field information into the position where the placeholder representing the field information is located in the above-mentioned database operation statement template, so as to update the database operation statement template.

[0052] The fourth sub-step is to determine the database operation type of the above-mentioned database operation statement template as the database operation type label information. For example, the above-mentioned database operation type label information can be one of the following: "insert operation", "update operation" and "delete operation".

[0053] The fifth sub-step is to determine the updated database operation statement template as the database operation statement information corresponding to the above-mentioned database operation type label information. As an example, the above-mentioned database operation statement information can be "INSERT INTO {table_transactions} (object_id, {fields}) VALUES ({000001}, {123456789,100})".

[0054] In some optional implementation manners of some embodiments, the above-mentioned off-site persistence node can perform batch database operations on the above-mentioned respective database operation statement information to update the above-mentioned information persistence database through the following steps:

[0055] The first step is to determine the database operation type label information corresponding to each database operation statement information in the above-mentioned respective database operation statement information as the target database operation type label information, where the above-mentioned target database operation type label information is one of the following: insert operation, update operation and delete operation.

[0056] The second step is to determine the above-mentioned respective database operation statement information as a set of database operation statement information.

[0057] The third step is to determine each database operation statement information with the target database operation type label information of "insert operation" in the above-mentioned set of database operation statement information as the first batch of database operation statement information group.

[0058] The fourth step is to determine each database operation statement information with the target database operation type label information of "update operation" in the above-mentioned set of database operation statement information as the second batch of database operation statement information group.

[0059] In the fifth step, determine the information of each storage operation statement whose target storage operation type label information in the above storage operation statement information set is a deletion operation as the third batch of storage operation statement information group.

[0060] In the sixth step, based on the above first batch of storage operation statement information group, the above second batch of storage operation statement information group, and the above third batch of storage operation statement information group, perform batch storage processing to update the above information persistent database.

[0061] In some optional implementation manners of some embodiments, the above off-site persistent node can perform batch storage processing based on the above first batch of storage operation statement information group, the above second batch of storage operation statement information group, and the above third batch of storage operation statement information group through the following steps to update the above information persistent database:

[0062] In the first step, perform parsing processing on each piece of first batch of storage operation statement information in the above first batch of storage operation statement information group to obtain parsing information, where the above parsing information includes table name information, column name group information, and value group information. In practice, the above off-site persistent node can perform parsing processing on the above first batch of storage operation statement information through regular expression technology to obtain table name information, column name group information, and value group information as parsing information. Optionally, the above off-site persistent node can perform parsing processing on the above first batch of storage operation statement information through an SQL parsing library to obtain table name information, column name group information, and value group information as parsing information. The above table name information can represent the name of the target table (such as transactions) into which the specified data will be inserted. The above column name group information can represent a column name group, and one column name in the above column name group can represent the column name of the column where the data is inserted into the list. The above value group information can represent a value group, that is, a data group inserted into the table. Each value in the above value group corresponds to one column name in the above column name group. As an example, the above first batch of storage operation statement information can be "INSERT INTO users (id, name) VALUES (1,‘Alice’)". The above column name group information can be "(id, name)", and the represented column name group can be "{id, name}". The above value group information can be "(1, ‘Alice’)", and the represented value group can be "{1, ‘Alice’}". Among them, the value "1" corresponds to the column name "id", and the value "‘Alice’" corresponds to the column name "name".

[0063] Step 2: Divide the parsing information with the same table name information and column name group information in each obtained parsing information into a group of parsing information as the target parsing information group, and obtain each target parsing information group. Among them, each target parsing information group in the above-mentioned target parsing information groups corresponds to group identification information, and the above-mentioned group identification information includes table name information and column name group information.

[0064] Step 3: For each target parsing information group in the above-mentioned target parsing information groups, perform the following generation steps:

[0065] The first sub-step: Determine the table name information and column name group information included in the group identification information corresponding to the above-mentioned target parsing information group as the table column name information to be inserted and the column name group information to be inserted.

[0066] The second sub-step: Merge and process each value group information included in the target parsing information group to obtain the value list information to be inserted. In practice, the above-mentioned off-site persistence node can connect each value group information through a preset symbol (such as a comma) to obtain the value list information to be inserted. As an example, the above-mentioned value group information can be "(1, 'Alice'), (2, 'tom')", then the above-mentioned value list information to be inserted can be "(1, 'Alice'), (2, 'tom')".

[0067] The third sub-step: Insert the above-mentioned table column name information to be inserted, the above-mentioned column name group information to be inserted, and the above-mentioned value list information to be inserted into the preset batch insertion template in batches to obtain the initial batch insertion statement information. For example, the above-mentioned insertion into the preset batch insertion template can be "INSERT INTO {table name} ({column name}) VALUES {value group}; where the value group is a comma-separated combination of multiple (value1, value2,...)". The above-mentioned table column name information to be inserted can be "table name: users, column name: (id, name)", and the value list information to be inserted can be "(1, 'Alice'), (2, 'tom')", then the above-mentioned initial batch insertion statement information can be "INSERT INTO users (id, name) VALUES (1, 'Alice'), (2, 'tom');"

[0068] Step 4: Determine each obtained initial batch insertion statement information as the batch insertion statement information.

[0069] Step 5: Send the above-mentioned batch insertion statement information to the above-mentioned information persistence database for the above-mentioned information persistence database to perform each batch insertion operation corresponding to the above-mentioned batch insertion statement information in parallel to update the data.

[0070] Step 6: In response to receiving the information sent by the information persistence database indicating that each batch insertion operation has been completed, generate batch update statement information based on the above second batch of database insertion operation statement information group. The information indicating that each batch insertion operation has been completed can be text information. For example, the information indicating that each batch insertion operation has been completed can be "Each batch insertion operation has been completed". In practice, the above off-site persistence node can generate batch update statement information based on the above second batch of database insertion operation statement information group through a preset tool (such as an ETL tool). The above batch update statement information can be an update SQL statement for performing batch update operations.

[0071] Step 7: Send the above batch update statement information to the above information persistence database for the above information persistence database to concurrently execute each batch update operation corresponding to the above batch update statement information.

[0072] Step 8: In response to receiving the information sent by the information persistence database indicating that the batch update operation has been completed, generate batch deletion statement information based on the above third batch of database insertion operation statement information group. The information indicating that the batch update operation has been completed can be text information. For example, the information indicating that the batch update operation has been completed can be "The batch update operation has been completed". In practice, the above off-site persistence node can generate batch deletion statement information based on the above third batch of database insertion operation statement information group through a preset tool (such as an ETL tool). The above batch deletion statement information can be an SQL statement for deleting multiple records at once.

[0073] Step 9: Send the above batch deletion statement information to the above information persistence database for the above information persistence database to concurrently execute the batch deletion operation corresponding to the above batch deletion statement information to update the data.

[0074] The above technical solution and its related content, as an inventive point of an embodiment of the present disclosure, solve the technical problem that "after generating the first batch of database insertion operation statement information groups, the second batch of database insertion operation statement information groups, and the third batch of database insertion operation statement information groups, traverse each database insertion statement information group (insertion, update, deletion), parse each SQL statement one by one and send it to the database for execution. Each database insertion operation (such as insertion, update, deletion) is usually executed one by one in the form of a single SQL statement, resulting in frequent establishment and destruction of database connections and a large database I / O overhead." The factors that lead to a large database I / O overhead are often as follows: after generating the first batch of database insertion operation statement information groups, the second batch of database insertion operation statement information groups, and the third batch of database insertion operation statement information groups, traverse each database insertion statement information group (insertion, update, deletion), parse each SQL statement one by one and send it to the database for execution. Each database insertion operation (such as insertion, update, deletion) is usually executed one by one in the form of a single SQL statement, resulting in frequent establishment and destruction of database connections and a large database I / O overhead. If the above factors are solved, the effect of reducing the database I / O overhead can be achieved. To achieve this effect, first, parse and process each first batch of database insertion operation statement information in the above first batch of database insertion operation statement information groups to obtain parsing information, where the above parsing information includes table name information, column name group information, and value group information. Thus, the respective parsing information for generating each target parsing information group can be obtained. After that, the parsing information with the same table name information and column name group information among the obtained respective parsing information is divided into a group of parsing information as a target parsing information group to obtain each target parsing information group, where each target parsing information group in the above respective target parsing information groups corresponds to group identification information, and the above group identification information includes table name information and column name group information. Then, for each target parsing information group in the above respective target parsing information groups, perform the following generation steps: The first sub-step is to determine the table name information and column name group information included in the group identification information corresponding to the target parsing information group as the table column names to be inserted and the column name group to be inserted. Thus, the table column names to be inserted and the column name group to be inserted for generating the initial batch insertion statement information can be used. The second sub-step is to merge the respective value group information included in the target parsing information group to obtain the value list information to be inserted. Thus, the value list information to be inserted for generating the initial batch insertion statement information can be obtained. The third sub-step is to insert the above table column names to be inserted, the above column name group to be inserted, and the above value list information to be inserted into the preset batch insertion template in batches to obtain the initial batch insertion statement information. Thus, the insertion operations represented by at least one first batch of database insertion operation statement information with the same table name can be merged to obtain the initial batch insertion statement information for batch insertion into a single table. Then, the obtained respective initial batch insertion statement information is determined as the batch insertion statement information.After that, the above batch insert statement information is sent to the above information persistence database for the above information persistence database to execute in parallel each batch insert operation corresponding to the above batch insert statement information to update the data. Thus, by executing in parallel each batch insert operation corresponding to the above batch insert statement information, multiple records can be inserted at one time, reducing the number of connection establishments and destructions, reducing the I / O overhead, and improving the execution efficiency. After that, in response to receiving the information sent by the above information persistence database indicating that each batch insert operation has been completed, batch update statement information is generated based on the above second batch of database write operation statement information group. Thus, after each batch insert operation has been completed, the multiple update operations represented by the second batch of database write operation statement information group can be merged into one batch update statement. The above batch update statement information is sent to the above information persistence database for the above information persistence database to execute the batch update operation corresponding to the above batch update statement information. Thus, the batch update statement information can be sent to the above information persistence database, causing the information persistence database to process multiple update instructions at one time, avoiding the performance overhead caused by executing one by one. Then, in response to receiving the information sent by the above information persistence database indicating that the batch update operation has been completed, batch delete statement information is generated based on the above third batch of database write operation statement information group. Thus, in the case where the batch update operation has been completed, the multiple delete operations represented by the third batch of database write operation statement information group can be merged into batch delete statement information representing one batch delete statement. Finally, the above batch delete statement information is sent to the above information persistence database for the above information persistence database to execute the batch delete operation corresponding to the above batch delete statement information to update the data. Thus, through the batch delete operation, records that meet the conditions can be removed at one time and the database can be updated, reducing the database performance overhead. At the same time, the batch insert, update, and delete operations are sequentially executed in order to ensure that there is data in the target table for subsequent operations, avoiding update or delete failures caused by data missing, and further reducing the possibility of deadlocks caused by the mixed execution of insert, update, and delete operations, resulting in mutual waiting for locks between transactions.

[0075] The above-described various embodiments of the present disclosure have the following beneficial effects: Through the user operation request information synchronization and batch persistence processing system of some embodiments of the present disclosure, network overhead, data synchronization latency, and hardware resource occupancy are reduced. Specifically, the reasons for the large network overhead, high data synchronization latency, and high hardware resource occupancy are as follows: An independent database is deployed in each region, and the user operation request information and operation return information within the region are synchronized to the central database through the built-in synchronization function of the database and third-party tools. Database synchronization of data usually requires frequent network interactions and transaction submissions, resulting in a large network overhead and a high data synchronization latency. At the same time, an independent database is deployed in each region, and the database instance needs to occupy an independent server and storage device, resulting in a high occupancy of hardware resources. Based on this, the user operation request information synchronization and batch persistence processing system of some embodiments of the present disclosure includes: business node clusters in each region, off-site persistence node clusters, and information persistence databases. Among them, the off-site persistence node clusters include each off-site persistence node, and each off-site persistence node in the above-mentioned off-site persistence nodes corresponds to one regional business node cluster in the above-mentioned business node clusters in each region. The regional business node clusters include a regional business master node and a regional business standby node. For each regional business node cluster in the above-mentioned business node clusters in each region, the following configurations are performed on the regional business node cluster: First, the regional business master node included in the regional business node cluster is configured to perform the following replication steps: The first step is to receive user operation requests and return information within a preset time, and generate a request and return message log file corresponding to the above-mentioned user operation requests and return information. Thus, a request and return message log file corresponding to the above-mentioned user operation requests and return information can be generated. The second step is to copy the above-mentioned request and return message log file to the regional business standby node included in the regional business node cluster. Thus, the request and return message log file can be copied to the regional business standby node. Thus, the user operation requests and return information can be collected by the regional business master node, and the request and return message log file can be copied to the regional business standby node. The above-mentioned regional business standby node is configured to perform the following synchronization steps: First, store the request and return message log file copied by the regional business master node in a preset stack. Thus, the request and return message log file can be stored in the preset stack so that when the main node log reporting service is interrupted later, the request and return message log file at the top of the stack can be retrieved in a timely manner. Then, monitor the log reporting status of the above-mentioned regional business master node to obtain log reporting status monitoring information. Thus, the log reporting status of the regional business master node can be monitored.After that, in response to determining that the log reporting status monitoring information characterizes the interruption of the main node log reporting service, the request and response message log file at the top of the above-mentioned preset stack is sent to the off-site persistent node corresponding to the above-mentioned regional service node cluster in the above-mentioned off-site persistent node cluster. Thus, when the main node log reporting service is interrupted, the request and response message log file at the top (i.e., the latest) can be synchronized to the off-site persistent node corresponding to the above-mentioned regional service node cluster in the above-mentioned off-site persistent node cluster. Thus, through the above-mentioned regional service standby node, when the main node log reporting service is interrupted, the request and response message log file at the top (i.e., the latest) can be synchronized to the off-site persistent node. The above-mentioned off-site persistent node is configured to perform the following batch persistence processing: First, based on the request and response message log file sent by the above-mentioned regional service standby node, each database operation statement information is generated. Thus, each database operation statement information for batch database storage processing of the user operation request and response information in the request and response message log file can be generated. Then, batch database storage processing is performed on the above-mentioned each database operation statement information to update the above-mentioned information persistent database. Thus, through the off-site persistent node, the information stored in the request and response message log file can be synchronized and persisted to the information persistent database (i.e., the central database). Also, because only the request and response message log file is transmitted during the process of synchronizing and persisting the received user operation request and response information to the information persistent database (i.e., the central database), there is no need for frequent network interaction and transaction submission, reducing network overhead and synchronization latency. At the same time, through the user operation request information synchronization and batch persistence processing system, the data user operation request information and operation response information collected by the regional service main node are synchronized to the information persistent database, without the need to deploy a complete database in each region, only logging and sending messages are required, and the hardware resource occupancy is less.

[0076] Figure 2 Flow 200 of some embodiments of the batch persistence processing of user operation request and response information of the off-site persistent node included in the above-mentioned user operation request information synchronization and batch persistence processing system according to the present disclosure is shown. The method for batch persistence processing of user operation request and response information includes the following steps:

[0077] Step 201, receive the request and response message log file sent by the regional service standby node.

[0078] In some embodiments, the execution subject for generating the building knowledge reply information (e.g., the off-site persistent node included in the user operation request information synchronization and batch persistence processing system) can receive the request and response message log file sent by the regional service standby node, where the above-mentioned request and response message log file contains each message data characterizing the user operation request and response information.

[0079] Step 202: Generate information on each database storage operation statement based on the request and response message log file.

[0080] In some embodiments, the above-mentioned execution entity may generate information on each database storage operation statement based on the above-mentioned request and response message log file.

[0081] Step 203: Perform batch database storage processing on the information on each database storage operation statement to update the information persistence database.

[0082] In some embodiments, the above-mentioned execution entity may perform batch database storage processing on the information on each database storage operation statement to update the above-mentioned information persistence database.

[0083] The above embodiments of the present disclosure have the following beneficial effects: Through the method for batch persistent processing of user operation request and return information applied to the off-site persistent node included in the above-mentioned user operation request information synchronization and batch persistent processing system in some embodiments of the present disclosure, the database load is reduced. Specifically, the reason for the relatively high database load is that: usually, the amount of log data contained in the request and response message log file is large. Directly writing all the log data (i.e., user operation request and return information) into the database at one time may cause the database load to be too high, and even lead to performance bottlenecks or outages. Based on this, the method for batch persistent processing of user operation request and return information applied to the off-site persistent node included in the above-mentioned user operation request information synchronization and batch persistent processing system in some embodiments of the present disclosure includes: First, receive the request and response message log file sent by the regional service standby node, where the above-mentioned request and response message log file contains each message data representing user operation request and return information. Thus, a request and response message log file containing each message data representing user operation request and return information can be received. After that, generate information on each database storage operation statement based on the above-mentioned request and response message log file. Thus, information on each database storage operation statement for batch database storage of the log data (i.e., user operation request and return information) included in the request and response message log file can be obtained. Finally, perform batch database storage processing on the information on each database storage operation statement to update the above-mentioned information persistence database. Performing batch database storage processing on the generated database storage operation statements avoids excessive pressure on the information persistence database caused by processing a large amount of data at one time, reduces the load of the information persistence database, and reduces the possibility of performance bottlenecks or outages.

[0084] The above description is only some preferred embodiments of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by specific combinations of technical features, but should also cover other technical solutions formed by any combination of technical features or their equivalent features without departing from the inventive concept. For example, a technical solution formed by mutually replacing features with technical features having similar functions (but not limited to) disclosed in the embodiments of the present disclosure.

Claims

1. A user operation request information synchronization and batch persistence processing system, characterized in that: The user operation request information synchronization and batch persistence processing system includes: business node clusters in each region, off-site persistence node clusters, and an information persistence database. Among them, the off-site persistence node clusters include each off-site persistence node, and each off-site persistence node in the off-site persistence nodes corresponds to one regional business node cluster in the regional business node clusters. The regional business node clusters include a regional business primary node and a regional business standby node; For each regional business node cluster in the regional business node clusters, the following configurations are performed on the regional business node cluster: The regional business primary node included in the regional business node cluster is configured to perform the following replication steps: Receive user operation requests and return messages within a preset time, and generate a request and return message log file corresponding to the user operation requests and return messages; Copy the request and return message log file to the regional business standby node included in the regional business node cluster; The regional business standby node is configured to perform the following synchronization steps: Store the request and return message log file copied from the regional business primary node into a preset stack; Monitor the log reporting status of the regional business primary node to obtain log reporting status monitoring information; In response to determining that the log reporting status monitoring information indicates that the primary node log reporting service is interrupted, send the request and return message log file at the top of the preset stack to the off-site persistence node corresponding to the regional business node cluster in the off-site persistence node cluster; The off-site persistence node is configured to perform the following batch persistence processing: Generate respective database operation statement information based on the request and return message log file sent by the regional business standby node; Perform batch database operations on the respective database operation statement information to update the information persistence database.

2. The user operation request information synchronization and batch persistent processing system according to claim 1, wherein: The regional business primary node is further configured to receive user operation requests and return messages within a preset time, and generate a request and return message log file corresponding to the user operation requests and return messages through the following steps: Receive each user operation request information and each operation return message as user operation requests and return messages from a preset data source within a preset time period; Convert the user operation requests and return messages into respective message data, where each message data in the respective message data corresponds to one piece of information in the respective user operation request information and the respective operation return messages; Store the respective message data into a preset log file; Determine the preset log file storing the respective message data as the request and return message log file corresponding to the user operation requests and return messages.

3. The user operation request information synchronization and batch persistent processing system according to claim 1, wherein: The regional business primary node is further configured to copy the request and return message log file to the regional business standby node included in the regional business node cluster through the following steps, including: Synchronize the request and return message log file to the regional business standby node included in the regional business node cluster through a pre-configured state machine replication mechanism.

4. The user operation request information synchronization and batch persistent processing system according to claim 1, wherein: The regional service standby node is further configured to monitor the log reporting status of the regional service primary node through the following steps to obtain log reporting status monitoring information, including: Receiving the log reporting status information sent by the regional service primary node; In response to determining that the log reporting status information indicates a log reporting failure, determining the information indicating the interruption of the primary node log reporting service as the log reporting status monitoring information.

5. The user operation request information synchronization and batch persistent processing system according to claim 2, wherein: The off-site persistence node is further configured to generate respective database operation statement information based on the request and return message log files sent by the regional service standby node through the following steps, including: Determining each message data included in the request and return message log file as respective message data to be converted, where each message data to be converted among the respective message data to be converted includes operation type information, primary key value information, and field information; For each message data to be converted among the respective message data to be converted, perform the following conversion steps: Determining the operation type information included in the message data to be converted as the operation type information to be queried; Querying the database operation statement template information corresponding to the operation type information to be queried from a preset operation statement mapping relation table to obtain the target database operation statement template information; Inserting the primary key value information and field information included in the message data to be converted into the database operation statement template corresponding to the target database operation statement template information to update the database operation statement template; Determining the database operation type of the database operation statement template as the database operation type label information; Determining the updated database operation statement template as the database operation statement information corresponding to the database operation type label information.

6. The user operation request information synchronization and batch persistence processing system according to claim 1, wherein: The off-site persistence node is further configured to perform batch database operations on the respective database operation statement information through the following steps to update the information persistence database, including: For each database operation statement information among the respective database operation statement information, determining the database operation type label information corresponding to the database operation statement information as the target database operation type label information, where the target database operation type label information is one of the following: insert operation, update operation, and delete operation; Determining the respective database operation statement information as a set of database operation statement information; Determining the respective database operation statement information with the target database operation type label information of insert operation in the set of database operation statement information as the first batch of database operation statement information group; Determining the respective database operation statement information with the target database operation type label information of update operation in the set of database operation statement information as the second batch of database operation statement information group; Determining the respective database operation statement information with the target database operation type label information of delete operation in the set of database operation statement information as the third batch of database operation statement information group; Performing batch database operations based on the first batch of database operation statement information group, the second batch of database operation statement information group, and the third batch of database operation statement information group to update the information persistence database.

7. A method for batch persistent processing of user operation requests and return information, which is applied to the remote persistent node included in the user operation request information synchronization and batch persistent processing system according to any one of claims 1-6. The method includes: Receiving a request and return message log file sent by a regional service backup node, where the request and return message log file contains each message data representing user operation requests and return information; Generating respective database insertion operation statement information based on the request and return message log file; Performing batch database insertion processing on the respective database insertion operation statement information to update the information persistent database.

Images