A distributed version management method, device and system

Abstract

The present invention discloses a distributed version management method, device and system, and relates to the field of computer technology. A specific implementation of the method includes: obtaining a message from a message queue; wherein the message includes a node identifier of the current business node and a local message version; according to the node identifier, intercepting a version field corresponding to the node identifier from the unified version information stored in the memory to obtain a global message version; wherein the unified version information includes version fields corresponding to messages of multiple business nodes, and multiple business nodes are used to complete the same business; comparing the local message version with the global message version, if the local message version is higher than the global message version, updating the unified version information and writing the message to the database. The method introduces a message queue under a distributed architecture, so that messages can be consumed serially, and the unlocking and unified management of message versions are realized, which reduces the database pressure to the greatest extent and improves the concurrent processing capability.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a distributed version management method, device and system. Background Art

[0002] For long-link services, their service data is usually assembled from messages of multiple service sub-processes. Since each service sub-process has its own independent state, the service data will be updated as the state of the service sub-process changes. Among them, long-link services refer to services that are completed by serializing and adding multiple links, and one link is used to implement one service sub-process of the service.

[0003] The above long-link business scenario has the characteristics of high message concurrency and the need to update data synchronously according to the status changes of the business sub-process. In order to reduce the disorder and repeated consumption of messages caused by high message concurrency and ensure that the status updates of multiple business sub-processes can be synchronously updated to the business data, it is necessary to perform full-process version management on the business data. When performing version management, the existing technology usually uses distributed locks and optimistic locks to achieve it.

[0004] In the process of implementing the present invention, there are at least the following problems in the prior art:

[0005] The version management method of the existing technology will cause lock contention in concurrent situations, consume lock resources, and affect system performance. At the same time, it relies on the database when updating data. A complete update process includes at least one database query operation and one database write operation, which puts a lot of pressure on the database. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a distributed version management method, device and system. The method introduces a message queue under a distributed architecture, so that message consumption can be serialized, and the message versions of messages generated by multiple business nodes are recorded through unified version information. When the message generated by the current business node is obtained, the message version carried by the message is first compared with the message version recorded in the unified version information. When it is determined that an update is required based on the comparison result, the message is written to the database, thereby achieving unlocking and unified management of message versions of multiple business nodes, avoiding disordered and repeated consumption messages, and achieving one-time access to the database, thereby minimizing the pressure of read and write operations on the database.

[0007] To achieve the above objective, according to one aspect of an embodiment of the present invention, a distributed version management method is provided.

[0008] A distributed version management method according to an embodiment of the present invention comprises: obtaining a message from a message queue; wherein the message is generated by the current business node executing a business sub-process, and comprises a node identifier and a local message version of the current business node; according to the node identifier, extracting a version field corresponding to the node identifier from unified version information stored in a memory to obtain a global message version; wherein the unified version information comprises version fields corresponding to messages of multiple business nodes, and the multiple business nodes are used to complete the same business; comparing the local message version with the global message version, and if the local message version is higher than the global message version, updating the unified version information and writing the message into a database; if the local message version is lower than or equal to the global message version, deleting the message.

[0009] Optionally, the version field is a binary value with a set step size; and according to the node identifier, the version field corresponding to the node identifier is intercepted from the unified version information stored in the memory to obtain the global message version, including: determining the position occupied by the version field corresponding to the message in the unified version information according to the node identifier and the step size; using bit operations to intercept the binary value of the position from the unified version information, and using the intercepted binary value as the global message version.

[0010] Optionally, the use of bit operation to extract the binary value of the position from the unified version information includes: generating a binary string; wherein the number of bits of the binary string is the same as the number of bits of the unified version information, and the value of the binary bit corresponding to the position in the binary string is 1, and the value of the remaining binary bits is 0; performing a bitwise AND operation on the unified version information and the binary string to obtain the binary value of the position.

[0011] Optionally, the message includes a primary key; before the step of extracting the version field corresponding to the node identifier from the unified version information stored in the memory, the method also includes: searching for the unified version information corresponding to the primary key according to the set association relationship, and determining that the search fails; wherein the association relationship is used to record the correspondence between the primary key and the unified version information; generating the unified version information according to the node identifier and the local message version, and storing the unified version information in the memory.

[0012] Optionally, generating the unified version information based on the node identifier and the local message version includes: converting the local message version into a corresponding binary value according to a set step size; filling the binary value into the version field corresponding to the node identifier, and filling the remaining version fields with 0 to obtain the unified version information.

[0013] Optionally, the message queue is used to store messages from the multiple service nodes with the same primary key; the messages are distributed to the message queue according to a preset hash function and the primary key.

[0014] Optionally, the method further comprises: synchronizing the updated unified version information to a remote database, and updating the offset of the message queue.

[0015] To achieve the above objective, according to another aspect of an embodiment of the present invention, a distributed version management device is provided.

[0016] A distributed version management device according to an embodiment of the present invention includes: a message acquisition module, which is used to obtain messages from a message queue; wherein the message is generated by the current business node executing a business sub-process, and includes the node identifier and local message version of the current business node; a version interception module, which is used to intercept the version field corresponding to the node identifier from the unified version information stored in the memory according to the node identifier, and obtain the global message version; wherein the unified version information includes version fields corresponding to messages of multiple business nodes, and the multiple business nodes are used to complete the same business; a comparison processing module, which is used to compare the local message version with the global message version, if the local message version is higher than the global message version, then update the unified version information and write the message to the database; if the local message version is lower than or equal to the global message version, then delete the message.

[0017] Optionally, the version field is a binary value with a set step size; the version interception module is also used to determine the position occupied by the version field corresponding to the message in the unified version information based on the node identifier and the step size; and use bit operations to intercept the binary value of the position from the unified version information, and use the intercepted binary value as the global message version.

[0018] Optionally, the version interception module is also used to generate a binary string; wherein the number of bits of the binary string is the same as the number of bits of the unified version information, and the value of the binary bit corresponding to the position in the binary string is 1, and the value of the remaining binary bits is 0; and the unified version information and the binary string are bitwise ANDed to obtain the binary value of the position.

[0019] Optionally, the message includes a primary key; the device also includes: a search processing module, used to search for the unified version information corresponding to the primary key according to the set association relationship, and determine that the search fails; wherein the association relationship is used to record the correspondence between the primary key and the unified version information; and based on the node identifier and the local message version, generate the unified version information, and store the unified version information in the memory.

[0020] Optionally, the search processing module is also used to: convert the local message version into a corresponding binary value according to a set step size; and fill the binary value into the version field corresponding to the node identifier, and fill the remaining version fields with 0 to obtain the unified version information.

[0021] Optionally, the message queue is used to store messages from the multiple service nodes with the same primary key; the messages are distributed to the message queue according to a preset hash function and the primary key.

[0022] Optionally, the device further comprises: an update synchronization module, configured to synchronize the updated unified version information to a remote database and update the offset of the message queue.

[0023] To achieve the above objective, according to another aspect of an embodiment of the present invention, a distributed version management system is provided.

[0024] A distributed version management system according to an embodiment of the present invention comprises: a distributed version management device and a message middleware; wherein the message middleware is used to receive messages from multiple business nodes, and distribute the messages to corresponding message queues according to a preset hash function and a primary key of the message.

[0025] Optionally, the hash function is used to perform a modulo operation on the primary key and the number of message queues of a set multiple.

[0026] To achieve the above objective, according to another aspect of an embodiment of the present invention, an electronic device is provided.

[0027] An electronic device according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement a distributed version management method according to an embodiment of the present invention.

[0028] To achieve the above objective, according to another aspect of the embodiments of the present invention, a computer readable medium is provided.

[0029] A computer-readable medium according to an embodiment of the present invention stores a computer program, and when the program is executed by a processor, a distributed version management method according to an embodiment of the present invention is implemented.

[0030] An embodiment of the above invention has the following advantages or beneficial effects: the message versions of messages generated by multiple business nodes are recorded through unified version information. When the message generated by the current business node is obtained, the message version carried by the message is first compared with the message version recorded in the unified version information. When it is determined that an update is required based on the comparison result, the message is written to the database, thereby achieving unlocking and unified management of message versions of multiple business nodes, avoiding disordered and repeated consumption messages, and realizing one-time access to the database, thereby minimizing the pressure of read and write operations on the database.

[0031] Using binary values ​​to construct unified version information, the version field corresponding to the node identifier in the unified version information can be intercepted through bit operations, and then the version is compared with the message version carried in the message, without the need to compare multiple versions with the database, to achieve version management. According to the determined position, a binary string with the same number of bits as the unified version information is generated, and then the binary string is bitwise ANDed with the unified version information to intercept the version field corresponding to the node identifier.

[0032] After obtaining the message, first search for the corresponding unified version information according to the primary key of the message. If the unified version information corresponding to the primary key cannot be found, it means that this is the first time to obtain a message with the primary key from the message queue. It is necessary to generate unified version information according to the node identifier and message version to facilitate version comparison when obtaining a new message. By converting the message version carried in the message into binary, the computing efficiency is improved.

[0033] By distributing messages from different business nodes to different message queues according to primary keys and hash functions, messages with the same primary key are sent to the same message queue. When consuming messages, only one machine can obtain the message, thus realizing serial processing of concurrent messages, avoiding resource competition, and improving the concurrent processing capability of the system. After the message is successfully written to the database, the updated unified version information is synchronized to the remote database, and the offset of the message queue is updated to ensure the consistency of the overall data and the high availability of the service.

[0034] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0036] Figure 1This is a schematic diagram of the structure of the wide table corresponding to the fulfillment data in the order business;

[0037] Figure 2 This is a schematic diagram of the principle of updating the business data of the order business;

[0038] Figure 3 is a schematic diagram of the main steps of the distributed version management method according to an embodiment of the present invention;

[0039] Figure 4 is a schematic diagram of the system architecture of a distributed version management method according to an embodiment of the present invention;

[0040] Figure 5 This is a schematic diagram of the main process of the distributed version management method according to an embodiment of the present invention;

[0041] Figure 6 2 is a schematic diagram of the updating principle of unified version information in the distributed version management method according to an embodiment of the present invention;

[0042] Figure 7 is a schematic diagram of main modules of a distributed version management device according to an embodiment of the present invention;

[0043] Figure 8 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0044] Fig. 9 It is a schematic diagram of the structure of a computer device suitable for implementing the electronic device of the embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0046] As described in the background technology, the business data of the long link business is assembled from messages of multiple business sub-processes. Specifically, the long link business is implemented by multiple business nodes. The data generated by each business node executing its own business sub-process is assembled into a wide table to obtain the corresponding business data. Among them, a wide table is a database table with more fields, usually referring to a database table associated with indicators, dimensions, and attributes related to the business subject.

[0047] Taking the order business as an example, the business nodes that implement this business include but are not limited to: order payment node, production node and distribution node. Among them, the order payment node is used to implement order placement, order payment, order splitting, order scheduling and other order sub-processes; the production node is used to implement the production sub-processes such as picking and packing; the distribution node is used to implement the distribution sub-process of goods delivery.

[0048] If you need to monitor the data of the entire link, such as analyzing the fulfillment data of the entire link and finding the fulfillment timeout information of certain business nodes, you need to store the relevant messages of the upstream and downstream of the order and perform aggregate statistical analysis. Among them, fulfillment data refers to the data generated during the entire life cycle from the generation of an order transaction to the end of the order transaction, including order data, production data, distribution data, etc.

[0049] Figure 1 This is the structure of the wide table corresponding to the fulfillment data of the order business. Figure 1 It can be seen that the wide table consists of order data fields, production data fields, and delivery data fields. The order data field includes information such as order creation time, payment time, order amount, and order status; the production data field includes information such as order download time, final picking time, picking completion time, packaging completion time, and picking list status; the delivery data field includes information such as the delivery person, delivery status, the user's expected latest delivery time, and self-pickup delivery identification.

[0050] Since the wide table of the long link service is processed by messages from multiple service sub-processes, the message of each service sub-process occupies part of the fields of the wide table. Figure 2 The wide table of the order business includes field1, field2, ..., field30 fields, which are processed by message 1 (including field1, field2, ..., field10 fields), message 2 (including field11, field12, ..., field20 fields) and message 3 (including field21, field22, ..., field30 fields).

[0051] in addition, Figure 2 In the example, ES is a search engine, and its full name is Elasticsearch. MySQL, ClickHouse, and HBase are all databases. status1 indicates that the message comes from the order payment node, status2 indicates that the message comes from the production node, and status3 indicates that the message comes from the distribution node.

[0052] Due to the uncertainty of the arrival time of the business sub-process messages, it will bring about high concurrency and disorder problems under distributed consumption. At the same time, since each business sub-process has its own independent state, the data update of the entire wide table will be updated as the state of the business sub-process changes, and multi-state management of the wide table data is required. For example, when the message of the production sub-process is updated, the wide table needs to be updated; when the message of the distribution sub-process is updated, the wide table also needs to be updated.

[0053] For the above specific business scenarios, it is necessary to properly handle the problems of disordered and repeated message consumption caused by high message concurrency to reduce the impact on the database; at the same time, it is also necessary to ensure that the status update of each business sub-process can be synchronized to the wide table to achieve version control of the entire process and prevent data status rollback. To achieve the above goals, the existing technology manages the status of wide table data by introducing a lock mechanism (including distributed locks and optimistic locks).

[0054] Among them, when the distributed lock method receives a message, it is necessary to first lock the primary key with a distributed lock, then query the status of the message in the database, compare the queried status with the status carried by the message, and write the message to the database when an update is required. When the optimistic lock method receives a message, it is necessary to first read the message (including the version number) from the database. When preparing to update, it is determined whether the current version number of the message in the database is consistent with the read version number. If the two are consistent, the version number is incremented and the message is written to the database.

[0055] However, the lock mechanism will cause lock competition (i.e., lock grabbing) in concurrent situations, consuming lock resources, with a high concurrent failure rate, requiring additional retries, and affecting system performance; at the same time, the data update depends on the database, and a complete update process includes at least one database query operation and one database write operation, which puts a lot of pressure on the database. Therefore, an embodiment of the present invention provides a distributed version management method, which realizes de-locking and avoids the above-mentioned problems, and is specifically implemented as follows.

[0056] Figure 3 FIG. 1 is a schematic diagram of the main steps of the distributed version management method according to an embodiment of the present invention. Figure 3 As shown, the distributed version management method of the embodiment of the present invention is implemented by a distributed version management device, and mainly includes the following steps:

[0057] Step S301: Get a message from the message queue. As the producer of the message, the current business node generates a message when executing its business sub-process. The message may include the node identifier of the current business node and the local message version. After the message is generated, it will be sent to the message middleware, which will store it in the message queue. Among them, the local message version refers to the version of the message generated by the current business node.

[0058] As a consumer of messages, the distributed version management device can obtain messages from the message queue based on the producer-consumer model or the publish-subscribe model. Taking the publish-subscribe model as an example, the distributed version management device subscribes to the messages of interest to the message middleware, and the message middleware then sends these messages to the distributed version management device.

[0059] Step S302: According to the node identifier, the version field corresponding to the node identifier is intercepted from the unified version information stored in the memory to obtain the global message version. The unified version information is stored in the memory, including version fields corresponding to messages of multiple business nodes, and multiple business nodes are used to complete the same business, such as completing the same order.

[0060] After the distributed version management device obtains the message, it intercepts the corresponding version field from the unified version information according to the node identifier carried in the message to obtain the global message version. The global message version is the version of the message generated by the current business node recorded in the unified version information, and the version occupies part of the fields of the unified version information.

[0061] Taking the order business as an example, this business is implemented by three business nodes, namely the order payment node, the production node and the distribution node. Therefore, the unified version information includes three version fields, namely the version field corresponding to the order payment node, the version field corresponding to the production node and the version field corresponding to the distribution node. Assuming that the order of the three version fields in the unified version information is the above order, if the node is identified as a production node, the second version field in the unified version information is intercepted.

[0062] Step S303: Determine whether the local message version is higher than the global message version. If the local message version is higher than the global message version, execute step S304; if the local message version is lower than or equal to the global message version, execute step S305. This step is used to determine whether the message version carried in the message is higher than the message version stored in the memory, and perform different processing according to the judgment result.

[0063] Step S304: Update the unified version information, write the message into the database, and end this process. If the message version carried in the message is higher than the message version stored in the memory, it means that the message needs to be updated, so the unified version information in the memory is updated, and then the message is written into the database.

[0064] When updating the unified version information in the memory, a new version field needs to be generated according to the message version carried in the message and the generation method of the version field in the unified version information, and the new version field replaces the intercepted version field to achieve the update of the unified version information.

[0065] Step S305: Delete the message and end this process. If the message version carried in the message is lower than the message version stored in the memory, it means that the message is an out-of-order message and can be directly deleted without updating the database. If the message version carried in the message is equal to the message version stored in the memory, it means that the message is a duplicate message and can be directly deleted without updating the database.

[0066] This embodiment provides a distributed version management method that is unlocked. By introducing a message queue under a distributed architecture, messages in the message queue can be consumed serially, avoiding resource consumption due to lock competition under high concurrency conditions. At the same time, this embodiment is also a version management method that merges multiple message versions, which realizes unified management of message versions carried by messages from different business nodes, avoiding the problems of disordered and repeated message consumption. In addition, this version management method is decoupled from the database, realizing one-time access to the database (each time a message is processed, the database is operated at most once), which minimizes the pressure of read and write operations on the database, and can speed up by more than twice compared to conventional locking mechanisms.

[0067] Figure 4 Schematic diagram of the system architecture of the distributed version management method according to an embodiment of the present invention. Figure 4 As shown, the distributed version management method of the embodiment of the present invention is implemented by a distributed version management system, which includes a message middleware and a distributed version management device. Among them, the message middleware connects to the message data streams of different business sub-processes. The functions implemented by the message middleware and the distributed version management device are specifically described below.

[0068] Message middleware is used to receive messages generated by multiple business nodes executing their respective business sub-processes, and distribute the messages to the corresponding message queues according to the primary key of the message and the set hash function.

[0069] In an embodiment, the hash function can be represented by Formula 1, which is used to perform a modulo operation on the primary key and the number of message queues times N. Preferably, the value of N is 1.

[0070] Index=(PK)%Num

[0071] Formula 1

[0072] In the formula, Index is the number of the message queue to which the message needs to be distributed; PK is the primary key of the message; Num is N times the number of message queues, where N is a positive integer; and % represents a remainder operation.

[0073] Furthermore, the message middleware is implemented based on Kafka. By applying for multiple partitions (i.e. a message queue), data is distributed to different partitions according to the primary key to ensure that the same primary key is sent to the same partition. When consuming messages later, you only need to serialize the messages without global locking.

[0074] A distributed version management device is used to subscribe to a single message queue. When a subscribed message is obtained from the message queue, the corresponding message version is intercepted from the unified version information stored in the memory according to the node identifier carried in the message. The message version carried in the message and the intercepted message version are then compared. When it is determined that the message needs to be updated (that is, the carried message version is higher than the intercepted message version), the unified version information is updated and the message is written to the database at one time.

[0075] In the embodiment, the message version may be a version number, message status information, etc. It is understandable that this embodiment supports transactional databases, such as Figure 4 In addition to ES and MySQL, it also supports OLAP databases and NOSQL databases. OLAP stands for Online Analytical Processing. OLAP databases include ClickHouse. NOSQL databases refer to non-relational databases, such as Redis and HBas e.

[0076] Figure 5 FIG. 1 is a schematic diagram of the main flow of the distributed version management method according to an embodiment of the present invention. Figure 5 As shown, the distributed version management method of the embodiment of the present invention mainly includes the following steps:

[0077] Step S501: Get a message from a message queue. The service of the distributed version management device acts as a consumer and consumes messages in one or more message queues. Messages with the same primary key are consumed by the same service. The message includes a primary key, a node identifier, and a local message version. Among them, the primary key is used to uniquely identify the message, the node identifier is the unique identifier of the business node that generates the message, and the local message version can be the status information of the message.

[0078] Step S502: according to the set association relationship, search for the unified version information corresponding to the primary key, if the search fails, execute step S503; if the search succeeds, execute step S504. The association relationship is used to record the corresponding relationship between the primary key and the unified version information.

[0079] Step S503: Generate unified version information according to the node identifier and local message version in the message, store the unified version information in the memory, and execute step S501. If the unified version information corresponding to the primary key is not found, it means that the message with the primary key is obtained from the message queue for the first time, and the unified version information corresponding to the business node needs to be generated.

[0080] In the embodiment, the unified version information includes the version fields corresponding to the messages of each service node that completes the same service. Each version field can be a decimal string or a binary string. The order of each version field can also be customized.

[0081] Taking the version field as a decimal string as an example, when generating unified version information, the local message version can be directly filled into the version field corresponding to the node identifier, and the remaining version fields are filled with 0 to obtain the unified version information.

[0082] Assume that service node 1, service node 2, and service node 3 complete an order together. When generating unified version information, the message version of service node 1 occupies the first position, the message version of service node 2 occupies the second position, and the message version of service node 3 occupies the third position. If the node identifier of the message in step S501 is service node 1 and the local message version is in state 1, the unified version information is 100.

[0083] Taking the binary value with a set step size as an example, when generating unified version information, you need to first convert the local message version into the corresponding binary value according to the step size, and then fill the binary value into the version field corresponding to the node identifier, and fill the remaining version fields with 0 to obtain the unified version information.

[0084] For example, the step size is set to 4, which means that a version field occupies 4 bits and can represent 16 states. Assuming that business node 1, business node 2, and business node 3 complete an order together, when generating unified version information, the message version of business node 1 occupies the first 4 bits, the message version of business node 2 occupies the middle 4 bits, and the message version of business node 3 occupies the last 4 bits. If the node identifier of the message in step S501 is business node 1 and the local message version is state 1, the corresponding binary value is 0001, and the unified version information is 000100000000.

[0085] In an optional embodiment, each version field may be separated by a separator, such as a space, a dash, etc. In addition, a temporary field may be added to the unified version information to add other required information according to business requirements.

[0086] Step S504: According to the node identifier, the version field corresponding to the node identifier is intercepted from the unified version information stored in the memory to obtain the global message version. If the unified version information corresponding to the primary key is found, the version field corresponding to the node identifier is intercepted from the found unified version information to obtain the global message version.

[0087] Taking the binary value with a set step size as an example, when intercepting the version field corresponding to the node identifier, you can first determine the position occupied by the version field corresponding to the message in the unified version information based on the node identifier and the step size; then use bit operations to intercept the binary value of this position from the unified version information, and use the intercepted binary value as the global message version.

[0088] In the embodiment, when the binary value of the position is intercepted from the unified version information, a binary string can be generated first, the number of bits of the binary string is the same as the number of bits of the unified version information, and the value of the binary bit corresponding to the position in the binary string is 1, and the value of the remaining binary bits is 0, and then the unified version information and the binary string are bitwise ANDed to obtain the binary value of the position. An example is given below.

[0089] Assume that the node identifier of the message in step S501 is business node 1, the local message version is in state 5, and the unified version information found is 000100000000. It can be determined that business node 1 occupies the first 4 bits of the unified version information, and then a binary string is generated, the first 4 bits of which are 1 and the last 8 bits are 0, that is, 111100000000; 111100000000 and 000100000000 are bitwise ANDed to obtain 0001, the decimal corresponding to which is 1, that is, the global message version is in state 1.

[0090] Step S505: Determine whether the local message version is higher than the global message version. If the local message version is higher than the global message version, execute step S506; if the local message version is lower than or equal to the global message version, execute step S508.

[0091] Step S506: Update the unified version information and write the message into the database. As in the above example, since the local message version is in state 5 and the global message version is in state 1, it can be seen that the local message version is higher than the global message version, the unified version information in the memory is updated, and the message is written into the database.

[0092] When updating the unified version information, the local message version can be converted into a corresponding binary value, and the converted binary value can be used to replace the intercepted version field to complete the update of the unified version information in the memory. In this example, the first 4 bits of the unified version information 0001 are replaced with 0101 to obtain the updated unified version information 010100000000.

[0093] Step S507: Synchronize the updated unified version information to the remote database, update the offset of the message queue, and end this process. After the message is successfully written to the database, synchronize the updated unified version information to the remote database (such as Redis), and then update the offset of the message queue. The offset is used to mark the location of the message. After the offset is updated, it means that the message consumption is successful and the latest message has been stored in the database.

[0094] This step can ensure the consistency of the overall data. If an exception occurs during any of the processes of writing to the database, synchronizing to the remote database, or updating the offset, the event will not be lost. The data can be consumed again by retrying, ensuring high availability of the service and consistency and security of the data.

[0095] Step S508: Delete the message and end this process. If the node identifier of the message in step S501 is business node 1, the local message version is in state 5, and the global message version corresponding to the business node intercepted in step S504 is also in state 5, it can be seen that the local message version is equal to the global message version, indicating that the obtained message is a duplicate message and is directly deleted.

[0096] If the node identifier of the message in step S501 is business node 1, the local message version is in state 5, and the global message version corresponding to the business node intercepted in step S504 is in state 6, it can be seen that the local message version is lower than the global message version, indicating that the obtained message is an out-of-order message and is directly deleted.

[0097] This embodiment uses binary values ​​to construct unified version information, so that version management can be achieved through bit operations. There is no need to pay attention to the data writing of the database in the future, and there is no need to compare multiple message versions with the database, which ensures the order of the overall message version and realizes the unified management of the message versions of messages generated by multiple business nodes. At the same time, batch processing can be achieved by pulling data from the message queue, and batch writing can be done at one time, further improving service performance.

[0098] In addition, in the above embodiment, no matter how many duplicate messages are obtained, the same processing method can be used to obtain the same processing result, which ensures the idempotence of the overall data; and the update of the offset is triggered only after the message is successfully written to the database and synchronized to the remote database, which ensures the transactionality of the overall data. Transactionality means that the operations included in the transaction are either all successful or all failed.

[0099] Figure 6 Schematic diagram of the updating principle of unified version information in the distributed version management method according to an embodiment of the present invention. Figure 6 As shown in the figure, service 1 subscribes to messages in message queue 1 and message queue 2, and service 2 subscribes to messages in message queue 3. status1 indicates that the message comes from business node 1 (such as the order payment node), and status2 indicates that the message comes from business node 2 (such as the production node). The message version is indicated by the status information.

[0100] Taking the example of service 1 obtaining a message from business node 1 from message queue 1, the message version carried in the message is state 1. Since this is the first time to obtain a message with this primary key from message queue 1, the memory does not store the unified version information of the business node. Therefore, the unified version information corresponding to the message of the business node is generated according to state 1.

[0101] At this time, the unified version information is 0001-0000-0000-0000. The first 4 bits of the unified version information are the version field corresponding to business node 1, the 5th to 8th bits are the version field corresponding to business node 2, the 9th to 12th bits are the version field corresponding to business node 3, and the last 4 bits are temporary storage bits. Then write this information to ES, synchronize the unified version information to Redis, and update the offset of message queue 1.

[0102] If a new message comes from business node 1, that is, service 1 obtains a message from business node 1 from message queue 1, and the message version carried in the message is state 5, the unified version information 0001-0000-0000-0000 can be obtained according to the primary key. After the message version is compared, the message version carried in the message is higher than the message version in the memory, so the unified version information is updated to 0101-0000-0000-0000. Then the information is written to ES, the updated unified version information is synchronized to Redis, and the offset of message queue 1 is updated.

[0103] Then, service 1 obtains the message from business node 2 from message queue 1. The message version carried in the message is state 2. Based on the primary key, the unified version information 0101-0000-0000-0000 can be obtained at this time. After comparing the message versions, the message version carried in the message is higher than the message version in the memory, so the unified version information is updated to 0101-0010-0000-0000. Then the information is written to ES, the updated unified version information is synchronized to Redis, and the offset of message queue 1 is updated.

[0104] After that, service 1 obtains a message from business node 1 from message queue 1. The message version carried in the message is in status 4. The unified version information 0101-0010-0000-0000 can be obtained based on the primary key. After comparing the message versions, the message version carried in the message is lower than the message version in the memory, indicating that the message is out of order and is directly discarded. The unified version information in the memory is still 0101-0010-0000-0000.

[0105] The following further describes the updating process of the unified version information in the embodiment of the present invention in conjunction with a specific business scenario.

[0106] Taking the order business as an example, its wide table data includes order data fields, production data fields, and delivery data fields. The primary key of each data field is the order number, and all are 100. The distributed version management device consumes each message in sequence according to the storage order of the messages in the message queue, and generates the corresponding unified version information. Table 1 shows the unified version information obtained after sequentially processing each message. Among them, orderStatus is the order status, wmsStatus is the production status, and tmsStatus is the delivery status.

[0107] Table 1 Example of unified version information

[0108]

[0109]

[0110] As can be seen from Table 1, the messages obtained by the distributed version management device from the message queue may be out of order, repeated, or order canceled. For the order cancellation scenario, when order data is received, if the local message version it carries is in state 10, indicating that the order is canceled, the version field corresponding to each business node can be directly set to all 1s. If a message with the primary key is received later, the unified version information will no longer change.

[0111] Figure 7 Schematic diagram of the main modules of the distributed version management device according to an embodiment of the present invention. Figure 7As shown, the distributed version management device 700 of the embodiment of the present invention mainly includes:

[0112] The message acquisition module 701 is used to obtain messages from the message queue. As a message producer, the current business node generates a message when executing its business sub-process. The message may include the node identifier and local message version of the current business node. After the message is generated, it will be sent to the message middleware, which will store it in the message queue.

[0113] As a consumer of messages, the distributed version management device can obtain messages from the message queue based on the producer-consumer model or the publish-subscribe model. Taking the publish-subscribe model as an example, the distributed version management device subscribes to the messages of interest to the message middleware, and the message middleware then sends these messages to the message acquisition module of the distributed version management device.

[0114] The version interception module 702 is used to intercept the version field corresponding to the node identifier from the unified version information stored in the memory according to the node identifier to obtain the global message version; wherein the unified version information includes version fields corresponding to messages of multiple business nodes, and the multiple business nodes are used to complete the same business.

[0115] The unified version information is stored in the memory, including the version fields corresponding to the messages of multiple business nodes, and multiple business nodes are used to complete the same business. After the distributed version management device obtains the message, it intercepts the corresponding version field from the unified version information according to the node identifier carried by the message to obtain the global message version.

[0116] The comparison processing module 703 is used to compare the local message version with the global message version. If the local message version is higher than the global message version, the unified version information is updated and the message is written into the database; if the local message version is lower than or equal to the global message version, the message is deleted.

[0117] The message version carried in the message is higher than the message version stored in the memory. If the message version carried in the message is higher than the message version stored in the memory, it means that the message needs to be updated. In this case, the unified version information in the memory is updated, and then the message is written to the database.

[0118] If the message version carried in the message is lower than the message version stored in the memory, it means that the message is out of order and can be deleted directly without updating the database. If the message version carried in the message is equal to the message version stored in the memory, it means that the message is a duplicate message and can be deleted directly without updating the database.

[0119] In addition, the distributed version management device 700 of the embodiment of the present invention may also include: a search processing module and an update synchronization module ( Figure 7 ). The search processing module is used to search for the unified version information corresponding to the primary key according to the set association relationship, and determine that the search fails; wherein the association relationship is used to record the correspondence between the primary key and the unified version information; and according to the node identifier and the local message version, the unified version information is generated and stored in the memory. The update synchronization module is used to synchronize the updated unified version information to the remote database and update the offset of the message queue.

[0120] From the above description, it can be seen that the message versions of messages generated by multiple business nodes are recorded through unified version information. When the message generated by the current business node is obtained, the message version carried by the message is first compared with the message version recorded in the unified version information. When it is determined that an update is required based on the comparison result, the message is written to the database. This realizes the unlocking and unified management of message versions of multiple business nodes, avoids disordered and repeated consumption messages, and realizes one-time access to the database, which minimizes the pressure of read and write operations on the database.

[0121] Figure 8 An exemplary system architecture 800 is shown to which the distributed version management method or distributed version management apparatus according to the embodiment of the present invention can be applied.

[0122] like Figure 8 As shown, system architecture 800 may include message middleware 801, network 802 and server 803. Network 802 is used to provide a medium for a communication link between message middleware 801 and server 803. Network 802 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0123] The message middleware 801 may be a server that provides various services, such as a proxy server that distributes messages generated by producers to different message queues according to primary keys.

[0124] Server 803 may be a server that provides various services, such as a message queue that subscribes to message middleware 801, or a background management server that processes messages. The background management server may obtain the global message version from the unified version information, compare the local message version carried by the message with the global message version, and perform different processing according to the comparison result.

[0125] It should be noted that the distributed version management method provided in the embodiment of the present application is generally executed by the server 803 , and accordingly, the distributed version management device is generally arranged in the server 803 .

[0126] It should be understood that Figure 8 The number of message middleware, networks and servers in the embodiment is only for illustration. Any number of message middleware, networks and servers may be provided according to implementation requirements.

[0127] According to an embodiment of the present invention, the present invention also provides an electronic device and a computer-readable medium.

[0128] The electronic device of the present invention comprises: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement a distributed version management method of an embodiment of the present invention.

[0129] The computer-readable medium of the present invention stores a computer program thereon, and when the program is executed by a processor, a distributed version management method of an embodiment of the present invention is implemented.

[0130] Reference below Fig. 9 , which shows a schematic diagram of the structure of a computer system 900 suitable for implementing an electronic device of an embodiment of the present invention. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0131] like Fig. 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the computer system 900 are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0132] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage section 908 as needed.

[0133] In particular, according to the embodiments disclosed in the present invention, the process described in the main step diagram above can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the main step diagram. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above-mentioned functions defined in the system of the present invention are executed.

[0134] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0135] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0136] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be arranged in a processor, for example, they may be described as: a processor including a message acquisition module, a version interception module and a comparison processing module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the message acquisition module may also be described as a "module for acquiring messages from a message queue".

[0137] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining a message from a message queue; wherein the message is generated by the current service node executing a service sub-process, including the node identifier and the local message version of the current service node; according to the node identifier, the version field corresponding to the node identifier is intercepted from the unified version information stored in the memory to obtain the global message version; wherein the unified version information includes the version fields corresponding to the messages of multiple service nodes, and the multiple service nodes are used to complete the same service; the local message version is compared with the global message version, and if the local message version is higher than the global message version, the unified version information is updated and the message is written into the database; if the local message version is lower than or equal to the global message version, the message is deleted.

[0138] According to the technical solution of the embodiment of the present invention, the message versions of messages generated by multiple business nodes are recorded through unified version information. When the message generated by the current business node is obtained, the message version carried by the message is first compared with the message version recorded in the unified version information. When it is determined that an update is required based on the comparison result, the message is written to the database. This realizes the unlocking and unified management of message versions of multiple business nodes, avoids disordered and repeated consumption messages, and realizes one-time access to the database, minimizing the pressure of read and write operations on the database.

[0139] The above product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0140] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A distributed version management method, characterized in that: include: Obtain a message from a message queue; wherein the message is generated by the current service node executing a service sub-process, and includes a node identifier and a local message version of the current service node; According to the node identifier, the version field corresponding to the node identifier is intercepted from the unified version information stored in the memory to obtain the global message version; wherein the unified version information includes version fields corresponding to messages of multiple service nodes, and the multiple service nodes are used to complete the same service; the version field is a binary value with a set step length; according to the node identifier, the version field corresponding to the node identifier is intercepted from the unified version information stored in the memory to obtain the global message version, including: determining the position occupied by the version field corresponding to the message in the unified version information according to the node identifier and the step length; using bit operation to intercept the binary value of the position from the unified version information, and using the intercepted binary value as the global message version; The local message version is compared with the global message version. If the local message version is higher than the global message version, the unified version information is updated and the message is written into the database. If the local message version is lower than or equal to the global message version, the message is deleted.

2. The method according to claim 1, characterized in that The using bit operation to extract the binary value of the position from the unified version information includes: Generate a binary string; wherein the number of bits of the binary string is the same as the number of bits of the unified version information, and the value of the binary bit corresponding to the position in the binary string is 1, and the value of the remaining binary bits is 0; A bitwise AND operation is performed on the unified version information and the binary string to obtain a binary value of the position.

3. The method according to claim 1, characterized in that The message includes a primary key; Before the step of intercepting the version field corresponding to the node identifier from the unified version information stored in the memory, the method further includes: According to the set association relationship, searching for the unified version information corresponding to the primary key, and determining that the search fails; wherein the association relationship is used to record the corresponding relationship between the primary key and the unified version information; The unified version information is generated according to the node identifier and the local message version, and the unified version information is stored in a memory.

4. The method according to claim 3, characterized in that The generating the unified version information according to the node identifier and the local message version includes: According to the set step size, convert the local message version into a corresponding binary value; The binary value is filled into the version field corresponding to the node identifier, and the remaining version fields are filled with 0 to obtain the unified version information.

5. The method according to claim 1, characterized in that The message queue is used to store messages from the multiple service nodes with the same primary key; the messages are distributed to the message queue according to a preset hash function and the primary key.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The updated unified version information is synchronized to a remote database, and the offset of the message queue is updated.

7. A distributed version management device, characterized in that: include: A message acquisition module, used to acquire a message from a message queue; wherein the message is generated by the current service node executing a service sub-process, and includes a node identifier and a local message version of the current service node; A version interception module is used to intercept the version field corresponding to the node identifier from the unified version information stored in the memory according to the node identifier to obtain a global message version; wherein the unified version information includes version fields corresponding to messages of multiple service nodes, and the multiple service nodes are used to complete the same service; the version field is a binary value with a set step length; according to the node identifier, the version field corresponding to the node identifier is intercepted from the unified version information stored in the memory to obtain the global message version, including: determining the position occupied by the version field corresponding to the message in the unified version information according to the node identifier and the step length; using bit operations to intercept the binary value of the position from the unified version information, and using the intercepted binary value as the global message version; A comparison processing module is used to compare the local message version with the global message version. If the local message version is higher than the global message version, the unified version information is updated and the message is written into the database; if the local message version is lower than or equal to the global message version, the message is deleted.

8. A distributed version management system, characterized in that: include: The distributed version management device and message middleware as claimed in claim 7; wherein, The message middleware is used to receive messages from multiple business nodes and distribute the messages to corresponding message queues according to a preset hash function and a primary key of the message.

9. The system according to claim 8, characterized in that The hash function is used to perform a modulo operation on the primary key and the number of message queues of a set multiple.

10. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

11. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images