Message redelivery method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a message redelivery method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of computers. According to the method, the first key value message is obtained by traversing the embedded database; the value of the first key value message comprises a delivery message, and the key of the first key value message comprises a first delivery timestamp and a first delivery frequency; the first key value message is delivered to the message-oriented middleware according to the first delivery timestamp; in response to delivery failure, updating the first delivery frequency to a second delivery frequency; determining a second delivery timestamp according to the second delivery times; generating a second key value message according to the second delivery timestamp, the second delivery times and the delivery message; and storing the second key value message as a new first key value message into the embedded database, and deleting the first key value message which fails to be delivered from the embedded database. According to the method and the device, the message can be quickly redelivered, the service timeliness is improved, and the working performance of the database is ensured not to be influenced by message delivery failure.

Description

Background Art

[0002] RocketMQ (message middleware) is a distributed message queue system for processing message delivery. The delivery device encapsulates the file data into messages and sends them to the message middleware for downstream receiving devices to process the data in parallel.

[0003] In related technologies, in order to improve the reliability of delivery, when delivery is abnormal, the message is written to a persistent storage such as a database, and then a scheduled task is used to compensate for the delivery. However, this solution can be used when the message middleware does not fail and a small number of messages fail to be delivered. However, when the message middleware fails and is unavailable as a whole, a large number of messages will fail to be delivered, and the delivery failure rate is high. Tens of millions of messages are inserted into the database slowly at the same time, resulting in a long time interval for re-delivery, and the downstream receiving device cannot receive messages for a long time, affecting the timeliness of the business. In addition, the database resources are occupied for a long time during the whole process, resulting in an overall decline in database performance.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present invention provides a message re-delivery method, device, electronic device, storage medium and program product to solve the problems of high message delivery failure rate, poor service timeliness and overall degradation of database performance.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a message re-delivery method is provided, comprising: traversing an embedded database to obtain a first key-value message, the value of the first key-value message comprising a delivery message, the key of the first key-value message comprising a first delivery timestamp and a first delivery count, and the delivery message is a message that fails to be delivered to a message middleware; delivering the first key-value message to the message middleware according to the first delivery timestamp; in response to a successful delivery, deleting the first key-value message from the embedded database; in response to a failed delivery, updating the first delivery count to a second delivery count; determining a second delivery timestamp according to the second delivery count; generating a second key-value message according to the second delivery timestamp, the second delivery count and the delivery message; storing the second key-value message as a new first key-value message in the embedded database, and deleting the first key-value message that fails to be delivered from the embedded database.

[0008] According to another aspect of the present disclosure, a message re-delivery device is also provided, including: a traversal module, used to traverse an embedded database to obtain a first key-value message; the value of the first key-value message includes a delivery message, and the key of the first key-value message includes a first delivery timestamp and a first delivery number; the delivery message is a message that fails to be delivered to the message middleware; a delivery module, used to deliver the first key-value message to the message middleware according to the first delivery timestamp; a response module, used to delete the first key-value message from the embedded database in response to a successful delivery; in response to a failed delivery, update the first delivery number to a second delivery number; according to the second delivery number, determine a second delivery timestamp; generate a second key-value message according to the second delivery timestamp, the second delivery number and the delivery message; store the second key-value message as a new first key-value message in the embedded database, and delete the first key-value message that fails to be delivered from the embedded database.

[0009] According to another aspect of the present disclosure, an electronic device is also provided, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the message re-projection methods described above by executing the executable instructions.

[0010] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the message re-direction method described in any one of the above is implemented.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including: a computer program or instructions, which implement any one of the above message re-direction methods when executed by a processor.

[0012] In an embodiment of the present disclosure, a first key-value message is obtained by traversing an embedded database; the value of the first key-value message includes a delivery message, and the key of the first key-value message includes a first delivery timestamp and a first delivery count; the delivery message is a message that fails to be delivered to the message middleware; and the first key-value message is delivered to the message middleware according to the first delivery timestamp; in response to a successful delivery, the first key-value message is deleted from the embedded database; in response to a failed delivery, the first delivery count is updated to a second delivery count; according to the second delivery count, a second delivery timestamp is determined; according to the second delivery timestamp, a second key-value message is generated; the second key-value message is stored in the embedded database as a new first key-value message, and the first key-value message that fails to be delivered is deleted from the embedded database. When the message middleware disclosed in the present invention is unavailable due to a failure, a large number of failed delivery messages can be quickly and persistently stored in an embedded database. The embedded database can convert random modifications of the delivery messages into sequential writes, so that tens of millions of messages can be quickly written into the embedded database, and can also be quickly queried from the embedded database, thereby achieving rapid re-delivery of messages and improving the timeliness of the business. By storing failed delivery messages in the embedded database, the database resources of the message middleware are not occupied, and the performance of the database work is guaranteed not to be affected by the failure of message delivery, thereby improving the database performance. In addition, re-delivering messages based on the delivery timestamp (such as the second delivery timestamp) determined by the number of delivery times can solve the problem of invalid delivery and reduce the failure rate of message re-delivery.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0015] Figure 1 A flow chart showing the method of compensating for delivery through timed tasks in the related art.

[0016] Figure 2 A schematic diagram showing the structure of a message re-direction system in an embodiment of the present disclosure.

[0017] Figure 3 A flow chart of a message re-delivery method in an embodiment of the present disclosure is shown.

[0018] Figure 4 A flow chart of a message re-direction method in another embodiment of the present disclosure is shown.

[0019] Figure 5 A flow chart of a message re-direction method in yet another embodiment of the present disclosure is shown.

[0020] Figure 6 A flow chart of a message re-direction method in another embodiment of the present disclosure is shown.

[0021] Figure 7 A diagram showing the message re-direction process in an embodiment of the present disclosure is shown.

[0022] Figure 8 A schematic diagram of a message re-projection device in an embodiment of the present disclosure is shown.

[0023] Fig. 9 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown.

[0024] Fig.10 A schematic diagram of a computer-readable storage medium provided in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0026] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0027] For ease of understanding, several terms involved in the present disclosure are explained below:

[0028] Message: A unit of data sent between two electronic devices. For example, the electronic devices may be computers.

[0029] Producer: The party that sends messages. The producer is responsible for creating messages and then sending them to the message middleware.

[0030] Receiving device (Consumer): responsible for subscribing to the topic in the message middleware and pulling messages from the subscribed topic.

[0031] RocketMQ (message middleware): is a distributed message middleware. It has the characteristics of low latency, high reliability, scalability, and ease of use. Producers write messages to the queue, and consumers take messages from the queue to perform business logic.

[0032] RocksDB (embedded database): RocksDB is an LSM-tree (Log-Structured Merge-Tree) architecture engine developed by Facebook based on LevelDB that provides key-value storage and read-write functions. The key-value pairs written by the user are first written to the WAL (Write Ahead Log) on ​​the disk, and then written to the SkipList (this part of the structure is also called MemTable) in the memory. The LSM-tree engine has a higher write throughput than the B-tree storage engine because it converts the user's random modifications (insertions) into sequential writes to the WAL file.

[0033] Thread: The corresponding English name is "thread", and its function is to execute a specific task in an application, such as a program, a function, etc.

[0034] Thread Suspend: Pause the thread execution.

[0035] Thread wakeup: resume thread running.

[0036] The following embodiments are described using a night-time batch collection system as an example, but the present disclosure is not limited thereto.

[0037] In the collection night batch system, it is necessary to parse the integrated files on the delivery device (such as the integrated files pushed by the upstream accounting system), and encapsulate the file data into messages and send them to the message middleware for other downstream nodes (receiving devices) to process the data in parallel. For example, in the collection night batch system, it is necessary to parse the contact, customer, and contract integrated files pushed by the upstream accounting system, and encapsulate the file data into message objects and send them to RocketMQ for downstream receiving devices to process the data in parallel. Due to the large amount of data at the level of tens of millions, delivery exceptions are often reported when delivered to the message middleware.

[0038] In related technologies, in order to solve the problem of delivery reliability, messages can be written to a persistent storage such as a database when delivery is abnormal, and then a scheduled task can be used to compensate for delivery. Figure 1 As shown, the solution specifically includes the following S101 to S105.

[0039] S101, parsing integrated file data.

[0040] S102, the producer creates a message and delivers the message to the message middleware. For example, the producer encapsulates file data into a message and delivers the message to the RocketMQ Broker server.

[0041] S103, the message middleware returns whether the message is delivered successfully. If it fails, execute S104. If it succeeds, end.

[0042] S104, writing the delivery failure message into a database, for example, into database table A.

[0043] S105, start the message re-drop service. It should be noted that the message re-drop service is an independent application (independent process) that reads data from the database regularly. If data is read, the data is encapsulated as a message and sent to the message middleware.

[0044] The above solution can be used when RocketMQ is not faulty and a small number of messages fail to be delivered. However, when RocketMQ fails and becomes unavailable as a whole, a large number of messages will fail to be delivered. Then the instantaneous writing pressure on the database will be very large. Writing to the database will take a long time, and a large number of messages written to the database may cause the database to be unavailable. After RocketMQ recovers from the failure, the failed messages need to be quickly re-delivered to the RocketMQ server, otherwise the timeliness of downstream consumption will be affected.

[0045] The inventors found that when RocketMQ is unavailable due to a failure, a large number of messages will fail to be delivered, and tens of millions of messages will be inserted into the database slowly at the same time, resulting in a long time between re-delivery. Downstream consumers cannot consume messages for a long time, affecting the timeliness of the business. In addition, the database resources are occupied for a long time during the whole process, causing the overall performance of the database to decline. In addition, the failed messages are concentratedly delivered to the message middleware, which easily leads to a large number of invalid messages being re-delivered, and a high failure rate of message delivery.

[0046] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0047] Based on this, the embodiments of the present disclosure provide a message re-delivery method, device, electronic device, storage medium and program product, which can be applied to the scenario where the message delivery fails and needs to be re-delivered. For example, RocksDB is used in the collection night batch system to delay and retry the integrated file delivery failure message to the message middleware. The present disclosure does not limit the specific application scenario. When the message middleware is unavailable due to a failure, a large number of failed delivery messages can be quickly and persistently stored in the embedded database. The embedded database can convert the random modification of the delivery message into a sequential write, so that tens of millions of messages can be quickly written into the embedded database, and can also be quickly queried from the embedded database, so as to achieve rapid re-delivery of messages and improve the timeliness of the business. By storing the failed delivery messages in the embedded database, the database resources of the message middleware are not occupied, and the performance of the database work is guaranteed not to be affected by the failure of message delivery, thereby improving the database performance. In addition, the re-delivery of the message according to the delivery timestamp (such as the second delivery timestamp) determined by the number of delivery times can solve the problem of invalid delivery and reduce the failure rate of message re-delivery.

[0048] It should be pointed out that, in the absence of conflict, the embodiments of the present disclosure and the technical features therein may be combined with each other.

[0049] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0050] Figure 2 A schematic diagram showing the structure of a message re-projection system in one embodiment of the present disclosure is shown. The system can apply the message re-projection method or message re-projection device in each embodiment of the present disclosure.

[0051] like Figure 2 As shown, the system architecture 200 may include a delivery device 201, a message middleware 202, a receiving device 203, an intermediate device 204, and a storage device 205. The delivery device 201 is used to deliver messages to the message middleware 202, the receiving device 203 is used to obtain messages from the message middleware 202, the intermediate device 204 may store the message that failed to be delivered in the form of a key-value pair (such as a second key-value message) in the embedded database thereon, and may also re-deliver the message stored in the embedded database to the message middleware 202, and the storage device 205 is provided with a database for providing a database function to the message middleware 202. For example, the storage device 205 is used to store data on the message middleware 202 and manage structured data.

[0052] In the disclosed embodiment, the embedded database and the database are deployed independently and on different devices (intermediate device and storage device), and do not occupy each other's hard disk resources, thereby achieving the storage of failed delivery messages through the embedded database without occupying the database resources of the message middleware, thereby ensuring that the performance of the database work is not affected by the failure of message delivery.

[0053] The delivery device 201, the message middleware 202 and the receiving device 203 are connected to each other through a network. The intermediate device 204 and the message middleware 202 are also connected to each other through a network. The storage device 205 and the message middleware 202 are also connected to each other through a network. The network can be a wired network or a wireless network.

[0054] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged through the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0055] The receiving device 203 is responsible for subscribing to the topic in the message middleware 202 and pulling messages from the subscribed topic. The receiving device 203 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, wearable devices, augmented reality devices, virtual reality devices, etc.

[0056] Optionally, the client of the application installed in different receiving devices 203 is the same, or is a client of the same type of application based on different operating systems. Based on different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.

[0057] The delivery device 201 is the party that sends the message. It can be responsible for creating a message and then sending it to the message middleware 202. The receiving device 203 receives the message from the message middleware 202 and consumes it. It should be noted that after the delivery device 201 fails to deliver a message to the message middleware 202, it can send the failed delivery message and the failed delivery return status (hereinafter referred to as fault) to the middleware 204.

[0058] The delivery device 201 may be a server or electronic device that provides various services. Exemplarily, the delivery device 201 may be a background management server that provides support for the device operated by the receiving device 203. The background management server may send the message that failed to be delivered to the message middleware 202 to the middleware 204, and the middleware 204 writes the delivered message in the form of a key-value pair to RocksDB, and re-delivers the message stored in RocksDB to the message middleware 202.

[0059] The intermediate device 204 and the storage device 205 may be servers that provide various services. The intermediate device 204 is installed with an embedded database that uses an LSM-tree storage engine, and the storage device 205 is installed with a database that provides regular operations and is not used to store redelivered data.

[0060] Optionally, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0061] Those skilled in the art will know that Figure 1 The number of delivery devices, message middleware, receiving devices, intermediate devices and storage devices in the embodiment is only illustrative, and any number of delivery devices, message middleware, receiving devices, intermediate devices and storage devices may be provided according to actual needs. The embodiments of the present disclosure do not limit this.

[0062] The present exemplary implementation is described in detail below with reference to the accompanying drawings and embodiments.

[0063] A message re-delivery method is provided in an embodiment of the present disclosure. The method can be executed by any electronic device with computing and processing capabilities, and can also be executed by the above-mentioned intermediate device.

[0064] Figure 3 A flow chart of a message retransmission method according to an embodiment of the present disclosure is shown as follows: Figure 3 As shown, applied in an intermediate device, the message re-direction method provided in the embodiment of the present disclosure includes the following S301 to S303.

[0065] S301, traverse the embedded database to obtain a first key-value message, the value of the first key-value message includes a delivery message, the key of the first key-value message includes a first delivery timestamp and a first delivery number, and the delivery message is a message that fails to be delivered to the message middleware.

[0066] The embedded database is a Key and Value storage engine based on the LSM-tree data structure. Since the LSM-tree engine converts the user's random modifications into sequential writes to the WAL file, it has a higher write throughput than the B-tree storage engine, supports high-throughput writes and fast range queries, and can complete the writing and reading of tens of millions of data into RocksDB in a very short time. The first key-value message in the disclosed embodiment is a key-value pair, which is stored in the embedded database.

[0067] In the embodiment of the present disclosure, the first delivery timestamp is used to indicate the time of redelivering the message. The embodiment of the present disclosure does not limit how the first delivery timestamp is represented. For example, the first delivery timestamp is 1615123200000. The first delivery timestamp represents time in milliseconds.

[0068] In the disclosed embodiment, the first delivery number is used to indicate the total number of times the same message is redelivered to the message middleware. For example, a delivery device delivers a message to the message middleware, and the message delivery fails. Since the delivery device delivers the message to the message middleware for the first time, the first delivery number is 0. For another example, if the delivery device fails to deliver a message to the message middleware, and the middleware redelivers the message to the message middleware, the first delivery number is updated to 1.

[0069] S302: Deliver a first key-value message to the message middleware according to the first delivery timestamp.

[0070] In the disclosed embodiment, when the timestamp on the intermediate device is greater than or equal to the first delivery timestamp, the first key-value message is delivered to the message middleware. For example, when the timestamp on the intermediate device is equal to the first delivery timestamp, the delivery message in the first key-value message is delivered to the message middleware.

[0071] S303, in response to successful delivery, delete the first key-value message from the embedded database; in response to failed delivery, update the first delivery number to the second delivery number; determine the second delivery timestamp according to the second delivery number; generate the second key-value message according to the second delivery timestamp, the second delivery number and the delivery message; store the second key-value message as the new first key-value message in the embedded database, and delete the first key-value message that failed to be delivered from the embedded database.

[0072] In the disclosed embodiment, if the delivery is successful, the first key-value message in the embedded database needs to be deleted to avoid repeated message delivery. If the delivery fails, the delivery time (such as the second delivery timestamp) needs to be updated, the delivery message is rewritten into the embedded database, and the first key-value message originally stored in the embedded database is deleted.

[0073] In one embodiment, in order to improve the accuracy of message delivery, the key of the key-value message may include a message identifier in addition to the delivery timestamp and the number of delivery times. The message identifier is an identifier used to indicate the uniqueness of the message, for example, the message identifier is a uuid (universally unique identifier).

[0074] It should be noted that, for the sake of convenience, the first key-value message and the second key-value message are collectively referred to as key-value messages. The first delivery timestamp and the second delivery timestamp are collectively referred to as delivery timestamps. The first delivery timestamp and the second delivery timestamp are collectively referred to as delivery times.

[0075] In another embodiment, when the message fails to be delivered to the message middleware, the return status is "fault", and the intermediate device can obtain the delivered message after receiving the fault. For example, the intermediate device obtains the delivered message from the delivery device. For another example, when the message is successfully delivered to the message middleware, the return status is "correct", and the intermediate device receives correct, it means that the delivery is normal, and the intermediate device does not need to obtain the delivered message.

[0076] It should be noted that as long as a message fails to be delivered to the message middleware, the message will be written into the embedded database in the form of a key-value pair, which is not affected by the re-delivery process. In other words, writing the failed delivery message into the embedded database in the form of a key-value pair and re-delivering the delivered message to the message middleware are two relatively independent threads.

[0077] For example, when a delivery device delivers a message to a message middleware and the message fails to be delivered, the delivery count of the message is set to 1, the delivery timestamp is determined based on the delivery count 1, the delivery message is used as the value of the key-value pair, and the delivery count and delivery timestamp are used as the keys of the key-value pair and written into the embedded database.

[0078] For another example, when the intermediate device delivers a delivery message to the message middleware (the delivery number of the delivery message in the key-value pair is 1), and the delivery message fails to be delivered, the delivery number of the delivery message is updated from 1 to 2, and the delivery timestamp is redetermined based on the delivery number 2, and the delivery message, delivery number and delivery timestamp are rewritten into the embedded database in the form of a key-value pair.

[0079] It should be noted that S301 to S303 of the present disclosure are re-investment processing threads. When there is a first key-value message in the embedded database, it means that a message delivery has failed and the re-investment processing thread needs to be awakened. The embodiment of the present disclosure can traverse the embedded database when receiving the message delivery failure information to detect whether there is a delivery message written to the database in the form of a key-value pair. If so, it detects whether the re-investment processing thread is turned on. If it is turned on, there is no need to reopen the re-investment processing thread. If not, the re-investment processing thread is turned on. In addition, the embodiment of the present disclosure can also traverse the embedded database at a set time in advance. For example, the embedded database is traversed every 5 seconds. If the first key-value message exists in the embedded database and the re-investment processing is not turned on, the re-investment processing is turned on.

[0080] In yet another embodiment, the message re-investment method may further include: when the re-investment processing thread is started, loopingly executing the re-investment processing thread.

[0081] In the disclosed embodiment, when the re-investment processing thread is executed, the re-investment processing will not be reopened even if a new key-value pair is written into the embedded database. The disclosed embodiment can maintain single-threaded operation (only one re-investment processing thread is opened), which can improve the stability of message re-investment.

[0082] In another embodiment, the message re-submission method may further include: when the number of first key-value messages in the embedded database is 0, closing the re-submission processing thread.

[0083] The disclosed embodiment is a closing condition of the reinvestment processing thread. When the reinvestment processing thread executes traversal of the embedded database, if the number of the first key-value messages traversed is 0, the reinvestment processing thread is closed. In other words, the reinvestment processing thread is suspended.

[0084] In the embodiment of the present disclosure, when the number of the first key value message in the embedded database is 0, it means that there is no message to be re-invested in the traversal of the embedded database, and the re-investment processing thread is suspended to wait for data. Suspending the re-investment processing thread can reduce the computing power of the operation, thereby saving costs and extending the life of the hardware.

[0085] When the message middleware disclosed in the present invention is unavailable due to a failure, a large number of failed delivery messages can be quickly and persistently stored in an embedded database. The embedded database can convert random modifications of the delivery messages into sequential writes, so that tens of millions of messages can be quickly written into the embedded database, and can also be quickly queried from the embedded database, thereby achieving rapid re-delivery of messages and improving the timeliness of the business. By storing failed delivery messages in the embedded database, the database resources of the message middleware are not occupied, and the performance of the database work is guaranteed not to be affected by the failure of message delivery, thereby improving the database performance. In addition, re-delivering messages according to the delivery timestamp determined by the number of delivery times can solve the problem of invalid delivery and reduce the failure rate of message re-delivery.

[0086] How to determine the delivery timestamp of a message according to the delivery timestamp will be described below through several exemplary embodiments.

[0087] In an exemplary embodiment, determining the second delivery timestamp according to the second delivery number may include the following steps A1 and A2.

[0088] Step A1, determining the delay time according to the second delivery number.

[0089] In the embodiment of the present disclosure, the larger the second delivery number, the longer the corresponding delay time. Regarding how to determine the delay time according to the second delivery number, the embodiment of the present disclosure does not specifically limit. For example, according to a preset relationship correspondence table, the delay time corresponding to the second delivery number is queried, and the relationship correspondence table is used to indicate the correspondence between the second delivery number and the delay time. The relationship correspondence table is shown in Table 1 below.

[0090] Table 1 Relationship correspondence table

[0091]

[0092] For another example, a line graph is used to indicate the corresponding relationship between the second delivery number and the delay duration. For another example, the second delivery number is input into a model (such as a neural network model) to obtain the corresponding delay duration.

[0093] In one embodiment, both the first delivery number and the second delivery number are exponentially related to the delay duration.

[0094] In the disclosed embodiment, the exponential relationship indicates that the relationship between two variables (number of deliveries and delay duration) is an exponential function relationship. When the number of deliveries and the delay duration are in an exponential relationship, as the number of deliveries increases, the delay duration will show an exponential growth, rather than a linear or other form of growth. This indicates that the delay duration increases faster and faster, rather than increasing at a fixed rate, thereby avoiding multiple invalid re-deliveries and improving the success rate of re-delivery.

[0095] In the disclosed embodiment, when the type of message middleware failure cannot be determined, the message middleware failure recovery time is uncertain. It may take a few minutes to recover, or several hours or days to recover. Therefore, the exponential backoff delayed delivery strategy is adopted to maximize the success rate of retry delivery and the utilization of system resources.

[0096] Step A2: determine a second delivery timestamp according to the delay duration.

[0097] In the disclosed embodiment, the second delivery timestamp may be the sum of the delivery failure time and the delay time. For example, if the message fails to be delivered at 2024-05-28 16:00, the delay time (e.g., 2 hours) is added to 2024-05-28 16:00 to obtain 2024-05-28 18:00, and this time (2024-05-28 18:00) is converted into a timestamp (i.e., the second delivery timestamp).

[0098] It should be noted that the recovery time of message middleware failures can range from several minutes to several hours. In the case of uncertainty about the recovery time of message middleware failures, a retry delivery strategy with exponential backoff of delay length is adopted. Once the failure is recovered, delivery will be completed in a short time to ensure the timeliness of the business.

[0099] In the embodiment of the present disclosure, the delay duration is determined by the number of deliveries, and the second delivery timestamp is determined by the delay duration, thereby ensuring the success rate of re-delivery and the utilization rate of system resources to the greatest extent.

[0100] In another exemplary embodiment, determining the delay duration according to the second delivery number may include the following steps B1 to B2.

[0101] Step B1, obtaining the fault type of the message middleware.

[0102] In the embodiments of the present disclosure, the message middleware has different fault types and different recovery times. For example, faults that can be recovered in a relatively short time include transient network failures, node restarts, transient resource exhaustion, and configuration errors. Faults that can be recovered in a relatively long time include hardware failures, serious network failures, and system upgrade failures. The embodiments of the present disclosure do not limit how to obtain the fault type of the message middleware. For example, the fault type is obtained by detecting the fault through a detection device. For another example, the message middleware reports the fault type.

[0103] Step B2, determining the delay time according to the fault type and the second delivery number. Wherein, when the second delivery number is the same, the longer the fault recovery time of the fault type is, the longer the delay time is determined.

[0104] Exemplarily, the fault type may include at least a first fault type and a second fault type, wherein the recovery time of the message middleware fault corresponding to the first fault type is shorter than the recovery time of the message middleware fault corresponding to the second fault type; when the second delivery number is the same, the delay period determined according to the first fault type is shorter than the delay period determined according to the second fault type.

[0105] In the disclosed embodiment, the relationship between the second delivery number and the delay time is determined according to the fault type of the message middleware. For example, for a short-term fault (such as the first fault type), the delay time is set to be relatively short. For another example, for a long-term fault (such as the second fault type), the delay time is set to be relatively long, thereby avoiding the failure of message delivery due to too many messages written into the message middleware instantaneously, improving the timeliness of receiving messages by the receiving device, and also improving the success rate of message delivery.

[0106] Exemplarily, the recovery time is determined according to the fault type, and before the recovery time, the delay time corresponding to the second delivery number is longer, and after the recovery time, the delay time corresponding to the second delivery number is shorter, so as to improve the success rate of message delivery while meeting the delivery timeliness.

[0107] For example, if the message middleware failure is a security incident, the message middleware server needs to be checked for security, patched for vulnerabilities, restored for data, and reinforced for the system. The actual recovery time is about 5 hours. At this time, the recovery time is set to 4 hours and 55 minutes. The current time is 12:00. Before 16:55, the relationship between the second delivery times and the delay duration is shown in Table 2 below.

[0108] Table 2 Relationship correspondence table

[0109]

[0110] After 16:55, the relationship between the second delivery number and the delay duration is shown in Table 3 below.

[0111] Table 3 Relationship correspondence table

[0112]

[0113] It should be noted that Table 2 and Table 3 of the disclosed embodiment only show the delay durations corresponding to some of the second delivery times. According to the actual application scenario and application experience, more delay durations corresponding to the second delivery times can be set. Based on Table 2 and Table 3, it can be seen that before the message middleware failure is restored, the message delay duration is long, which reduces a large number of invalid deliveries. When the message middleware failure is about to be restored, the message delay duration is short, which can increase the timeliness of re-delivery.

[0114] In the embodiment of the present disclosure, the delay duration is determined by the second delivery number and the fault type, thereby ensuring the success rate of redelivery and the utilization rate of system resources to the greatest extent.

[0115] In yet another exemplary embodiment, Figure 4 As shown, determining a second delivery timestamp according to the second delivery number includes S401 to S403.

[0116] S401, obtaining the average load occupancy rate of the message middleware within the sliding time window.

[0117] In the disclosed embodiments, the average load utilization of the message middleware generally refers to the ratio between the load processed by the message middleware system and the maximum processing capacity of the system within a sliding time window.

[0118] S402: When the average load occupancy rate is less than a preset load occupancy threshold, determine the delay duration according to a preset delay duration initial value, a preset delay duration increment and a second delivery number.

[0119] In the embodiment of the present disclosure, the specific value of the preset load occupancy threshold is not limited in the embodiment of the present disclosure. For example, the preset load occupancy threshold is any one of 90%, 85%, 80%, 75% and 70%.

[0120] Exemplarily, determining the delay duration based on a preset delay duration initial value, a preset delay duration increment, and a second number of deliveries may include: calculating the delay duration increment raised to the power of the second number of deliveries to obtain a first value; and calculating the product of the delay duration initial value and the first value to obtain the delay duration.

[0121] The calculation formula of the delay time is shown in the following formula 1.

[0122] B1=I×R c (1)

[0123] Among them, B1 is the delay duration, I is the initial value of the delay duration, R is the preset delay duration increment, and c is the second delivery number.

[0124] S403: Determine a second delivery timestamp according to the delay duration.

[0125] In the embodiment of the present disclosure, how to determine the second delivery timestamp according to the delay duration has been described in the above exemplary embodiments and will not be repeated here.

[0126] It should be noted that when the average load occupancy rate is less than the preset load occupancy threshold, it means that the message middleware can process the delivery message at a normal rate.

[0127] The disclosed embodiment can further improve the success rate of message delivery by dynamically adjusting the delay duration of message delivery according to the message re-delivery situation (number of deliveries) and the current message middleware load situation.

[0128] In another exemplary embodiment, after obtaining the average load occupancy rate of the message middleware in the sliding time window, the method provided by the embodiment of the present disclosure may further include the following steps C1 and C2.

[0129] Step C1, when the average load occupancy rate is greater than or equal to the preset load occupancy threshold, obtain the total number of successful deliveries in the sliding time window, and obtain the average processing time based on the total number of successful deliveries in the sliding time window and the total time of the sliding time window; determine the delay duration based on the preset delay duration initial value, the preset delay duration increment, the second number of deliveries, the average processing time and the average load occupancy rate.

[0130] In the embodiment of the present disclosure, the specific value of the preset load occupancy threshold is not limited in the embodiment of the present disclosure. For example, the preset load occupancy threshold is any one of 90%, 85%, 80%, 75% and 70%. For another example, the preset load occupancy threshold is 80%.

[0131] It should be noted that when the average load occupancy rate is greater than or equal to the preset load occupancy threshold, it can be confirmed that the message middleware is in a high load state and the delay time needs to be increased to reduce the number of message delivery failures.

[0132] In the disclosed embodiment, when the average load occupancy rate is greater than or equal to a preset load occupancy threshold, the delay duration needs to be adjusted to slow down the delivery speed of the message, thereby increasing the success rate of the delivery.

[0133] In one embodiment, the delay duration is determined according to a preset delay duration initial value, a preset delay duration increment, a second delivery number, an average processing time, and an average load occupancy rate, including steps C11 to C15.

[0134] Step C11, calculating the delay time increment raised to the power of the second number of deliveries to obtain a second value.

[0135] Step C12, calculating the product of the initial value of the delay time and the second value to obtain a third value.

[0136] In the embodiment of the present disclosure, the third value can be calculated by formula 1. That is, the third value in the embodiment of the present disclosure is B1 in formula 1.

[0137] Step C13, calculating the ratio of the average processing time to the total time of the sliding time window to obtain a fourth value.

[0138] In the disclosed embodiment, the average processing time is the ratio of the total time of the sliding time window to the total number of successful message delivery times.

[0139] Step C14, calculating the ratio of the average load occupancy rate to the preset load occupancy threshold to obtain a fifth value.

[0140] In the embodiment of the present disclosure, the average load occupancy rate is the average system load in the sliding time window. The embodiment of the present disclosure does not specifically limit how to calculate the average load occupancy rate. For example, the total time of the sliding time window is 5 minutes, and the load occupancy rates obtained at 1 minute, 2 minutes, 3 minutes, 4 minutes and 5 minutes are 80%, 90%, 70%, 80% and 80% respectively. The average load occupancy rate is obtained by calculating the average value, and the calculated average load occupancy rate is 80%.

[0141] Step C15, determining the delay duration based on the third value, the fourth value and the fifth value.

[0142] In the embodiment of the present disclosure, when one or more of the third value, the fourth value and the fifth value increases, the value of the delay duration also increases accordingly.

[0143] In one embodiment, the delay duration is determined by the following formula 2.

[0144]

[0145] Wherein, B2 is the delay duration, I is the initial value of the delay duration, R is the preset delay duration increment, c is the second delivery number, T 均 is the average processing time, T 总 is the total time of the sliding time window, F 均 is the average load occupancy, F 阈 It is the preset load occupancy threshold.

[0146] It should be noted that I×R c is the third value, is the fourth value, The fifth value.

[0147] In another embodiment, the delay duration is determined by the following formula 3.

[0148]

[0149] Wherein, B2 is the delay duration, I is the initial value of the delay duration, R is the preset delay duration increment, c is the second delivery number, T 均 is the average processing time, T 总 is the total time of the sliding time window, F 均 is the average load occupancy, F 阈is the preset load occupancy threshold, and W is the adjustment coefficient.

[0150] In the embodiment of the present disclosure, when the average load occupancy rate is greater than or equal to a preset load occupancy threshold, the delay duration is adjusted. The greater the load occupancy rate, the longer the delay time, thereby increasing the success rate of delivery.

[0151] Step C2, determining a second delivery timestamp according to the delay duration.

[0152] In the embodiment of the present disclosure, how to determine the second delivery timestamp according to the delay duration has been described in the above exemplary embodiments and will not be repeated here.

[0153] The disclosed embodiment can further improve the success rate of message delivery by dynamically adjusting the delay duration of message delivery according to the message re-delivery situation (number of deliveries) and the current message middleware load situation.

[0154] In yet another exemplary embodiment, determining the delay duration based on the third value, the fourth value, and the fifth value may include steps D1 to D4.

[0155] Step D1, obtaining a candidate delay duration based on the third value, the fourth value and the fifth value.

[0156] In the embodiment of the present disclosure, the candidate delay duration can be calculated by Formula 2 or Formula 3. That is, the candidate delay duration in the embodiment of the present disclosure is B2 in Formula 2 or Formula 3.

[0157] Step D2, obtaining a preset delay time threshold.

[0158] In the embodiment of the present disclosure, the delay duration threshold is the maximum delay duration. The maximum delay duration can be set according to actual application scenarios and specific application experience.

[0159] Step D3: when the candidate delay duration is less than or equal to the delay duration threshold, determine the candidate delay duration as the delay duration.

[0160] Step D4: when the candidate delay duration is greater than the delay duration threshold, the delay duration threshold is determined as the delay duration.

[0161] In order to avoid the delay duration from increasing indefinitely, the embodiment of the present disclosure compares the candidate delay duration with the delay duration threshold, and selects the smaller value between the two as the delay duration, thereby ensuring the timeliness of message delivery.

[0162] The above describes how to determine the delivery timestamp. The following describes how to resubmit a message and how to handle the resubmission result.

[0163] In an exemplary embodiment, delivering a first key-value message to a message middleware according to a first delivery timestamp may include: splitting the key of the first key-value message to obtain a first delivery timestamp and a first delivery number of the delivery message; and delivering the delivery message to the message middleware when the current time is greater than or equal to the first delivery timestamp and the first delivery number is less than a preset delivery number.

[0164] In the disclosed embodiment, the method for splitting the key in the first key-value message is not limited. For example, the key is split by the string processing function Split to obtain the first delivery timestamp and the first delivery number of the delivery message.

[0165] Exemplarily, the embedded database is traversed to obtain multiple first key-value messages, and the KEY of a first key-value message is taken out from the multiple first key-value messages, where the KEY is [first delivery timestamp, first delivery number], and the string processing function Split ("|") is used to split the KEY to obtain the first delivery timestamp and the first delivery number.

[0166] In the embodiment of the present disclosure, the current time may be a current timestamp of the intermediate device, or may be a 24-hour time corresponding to the current timestamp of the intermediate device.

[0167] In the embodiment of the present disclosure, there are two conditions for redelivering a delivered message (i.e., a time condition and a number condition). When the current time is greater than the first delivery timestamp, it means that the delivered message has reached the delivery time, that is, the time condition is met. When the first delivery number is less than the preset delivery number, it means that the number condition is met. When both conditions are met, the delivered message is redelivered to the message middleware. The embodiment of the present disclosure does not limit the specific value of the preset delivery number. For example, the preset delivery number is any value from 5 to 20. Exemplarily, the preset delivery number is any value from 5, 6, 7, 8, 9, and 10.

[0168] It should be noted that when the first delivery number is greater than or equal to the preset delivery number, no delivery will be performed.

[0169] In the embodiment of the present disclosure, when the current time is greater than or equal to the first delivery timestamp and the first delivery number is less than the preset delivery number, the message is re-delivered to the message middleware, thereby ensuring the success rate of the re-delivery and the utilization rate of system resources to the greatest extent.

[0170] In another exemplary embodiment, Figure 5 As shown, after delivering the first key-value message to the message middleware according to the first delivery timestamp, the method may further include S501 to S503.

[0171] S501, determining whether the message is successfully re-delivered to the message middleware.

[0172] In the embodiments of the present disclosure, whether the re-delivery of a delivered message is successful can be determined by the return status of the message delivery. If the return status is "correct", the message re-delivery is successful. If the return status is "fault", the message re-delivery fails. It should be noted that the embodiments of the present disclosure do not limit the type of flag used to identify the return status, as long as it can indicate the success or failure of the delivery.

[0173] S502: In response to successful delivery, delete the first key-value message corresponding to the message from the embedded database.

[0174] In the embodiment of the present disclosure, in order to avoid duplicate delivery, the first key-value message corresponding to the successfully delivered message is deleted from the embedded database.

[0175] S503, in response to a delivery failure, updating the first delivery number to a second delivery number; determining a second delivery timestamp based on the second delivery number; generating a second key-value message based on the second delivery timestamp, the second delivery number and the delivery message; storing the second key-value message as a new first key-value message in an embedded database, and deleting the first key-value message that failed to be delivered from the embedded database.

[0176] In the disclosed embodiment, as the number of delivery times is updated, the number of delivery times gradually increases. Exemplarily, after the intermediate device fails to deliver the first key-value message to the message middleware, the value of the number of delivery times corresponding to the message in the first key-value message is increased by 1, and the second delivery timestamp is determined based on the number of delivery times after the increase by 1, and the second delivery timestamp, the second delivery timestamp, the second delivery timestamp and the delivery message are written into the embedded database in the form of a key-value pair to obtain the second key-value message, and the first key-value message is deleted from the embedded database, that is, the original first key-value message corresponding to the delivery message in the embedded database is deleted.

[0177] Exemplarily, if it is determined that the current time is greater than the first delivery timestamp, and the first delivery number is less than the preset delivery number (such as the maximum delivery number), the delivery message is delivered to the message middleware. If the return status of the message middleware is successful delivery, the first key-value message corresponding to the message is deleted from the embedded database, which can be deleted by calling the API delete method. If the return status is failure, the delivery number of the current message is increased by 1, and the second delivery timestamp is recalculated according to the delivery number, the delivery message is copied, and the new KEY is used to rewrite RocksDB, where the format of KEY is [first delivery timestamp, first delivery number], and finally the original first key-value message is deleted. Then continue to take the next first key-value message until all the traversed first key-value messages are re-delivered.

[0178] It should be noted that the number of first key-value messages traversed from the embedded database can be 0, 1, or multiple. Exemplarily, when the number of first key-value messages traversed from the embedded database for the first time is multiple, the multiple first key-value messages can be regarded as a key-value data set D0. When the key-value data set D0 is processed, the first key-value messages in RocksDB are traversed from the beginning to obtain the next batch of key-value data sets D1. If the key-value data set D1 is not an empty set, the re-delivery is continued. If it is an empty set, the re-delivery processing thread is suspended to wait for data.

[0179] When re-delivery fails, the disclosed embodiment updates the first delivery number and the first delivery timestamp, and re-writes the unsuccessfully delivered message into the embedded database in the form of a key-value pair, thereby ensuring the timeliness of the delivery of other first key-value messages in the embedded database, and also ensuring that the delivered message is not duplicated, thereby improving the accuracy of business services.

[0180] In order to further improve the non-duplication of messages delivered to the message middleware, the present disclosure provides the following exemplary embodiments.

[0181] In an exemplary embodiment, a message identifier is added to the key of the first key-value message, where the message identifier is an identifier used to indicate the uniqueness of the message.

[0182] Exemplarily, the embedded database is traversed to obtain multiple first key-value messages, and the KEY of a first key-value message is taken out from the multiple first key-value messages, where the KEY is [first delivery timestamp, first delivery number, uuid]. The string processing function Split (“|”) is used to split the KEY to obtain the first delivery timestamp, first delivery number and uuid, where uuid is the message identifier.

[0183] When re-delivering a message, in addition to meeting the time condition and the number condition, the disclosed embodiment also needs to determine whether the message identifier of the delivered message is the same as the message identifier of the previously delivered message. If they are different, the delivered message is re-delivered according to the first delivery timestamp. If they are the same, the delivered message is not delivered, and the first key value message corresponding to the delivered message is deleted from the embedded database.

[0184] The disclosed embodiment can ensure that the delivered messages are not repeated by judging whether the message identifiers are the same. If the delivered messages are not repeated, the receiving device will not consume the messages repeatedly, thereby further improving the accuracy of the business service.

[0185] In order to solve the problem of large disk memory usage due to failure to deliver messages in the embedded database, such as Figure 6 As shown, the message re-delivery method may further include S601 to S603.

[0186] S601, traverse the embedded database at a preset cleaning time to obtain a first key-value message.

[0187] In the embodiment of the present disclosure, the preset cleaning time can be the time when the scheduled task polling (also known as the cleaning data scheduled task) starts, for example, traversing the embedded database at 12:00 every night to obtain the first key-value message. For another example, traversing the embedded database at 12:00 every Monday night to obtain the first key-value message. The present disclosure is not limited to this and can be set according to actual application scenarios and specific application experience.

[0188] It should be noted that when the embedded database is traversed at the preset cleaning time, the number of first key value messages traversed is 0, indicating that there is no need to clean up the disk memory.

[0189] S602: Obtain the delivery failure time of the first key-value message.

[0190] In the disclosed embodiment, the delivery failure time can be represented by a delivery failure timestamp, or can be represented by a 24-hour time. The delivery failure time is the time or timestamp of the most recent delivery failure of the delivery message. For example, if the delivery message fails at 2024-05-28 16:00, assuming that this is the last delivery and the delivery fails, the timestamp converted to 2024-05-28 16:00 is the delivery failure timestamp.

[0191] S603: If the difference between the current time and the delivery failure time is greater than the preset delivery duration, the first key-value message is deleted from the embedded database.

[0192] In the embodiment of the present disclosure, the preset delivery time can be set as needed. For example, the preset delivery time is 7 days, and the delivery failure time is 2024-05-28 16:00, and a scheduled task polling is set to delete the first key value message 7 days after the delivery failure time.

[0193] The disclosed embodiment can avoid the problem of data occupying the disk for a long time by deleting delivery messages that cannot be delivered for a long time, thereby improving work efficiency and reducing energy consumption.

[0194] In summary, the present disclosure re-delivers the delivered message to the message middleware based on RocksDB. When the message middleware fails and is unavailable, it can quickly persist tens of millions of large data messages to the disk, and if the message middleware fails for a short time and cannot be restored, it supports customized delayed delivery of messages. After the failure of the message middleware is resolved, the failed message can be delivered to the message middleware in a short time. Compared with MySQL (open source relational database management system) to store the failed delivery message, it can greatly reduce the time for re-delivery, does not occupy database resources, and solves the problem of reliable message delivery.

[0195] The present disclosure is described below through complete examples.

[0196] like Figure 7 As shown, combined Figure 7 , describes the specific implementation of the instance according to the direction of the data flow. It should be noted that the direction of the data flow has been Figure 7 Marked in.

[0197] (1) The delivery device parses and integrates the file data.

[0198] (2) The delivery device encapsulates the file data into a message.

[0199] (3) The delivery device delivers messages to the message middleware.

[0200] (4) Determine whether the message is delivered successfully based on the return status of the message middleware. If the message is delivered successfully, then end. If the message fails to be delivered, execute step (5).

[0201] (5) In response to a failure in message delivery to the message middleware, determine the delivery message, the first delivery number and the message identifier, determine a first delivery timestamp based on the first delivery number, use the delivery message as the value of the first key-value message, use the first delivery timestamp, the first delivery number and the message identifier as the key of the first key-value message, and write them into the embedded database.

[0202] (6) When the embedded database has the first key value message and the reinvestment process is not enabled, the reinvestment process thread is awakened. Exemplarily, the reinvestment process thread can be awakened by calling the thread start method, and the embedded database is polled (for example, the disk directory of the embedded database is polled). If there is no first key value message, the reinvestment process thread is suspended by calling the thread Thread.wait method.

[0203] (7) The intermediate device prepares to resubmit the message to the message middleware. In other words, it prepares to execute the resubmission process in a loop, and the resubmission process may include: traversing the embedded database to obtain the first key-value message, the value of the first key-value message includes the delivery message, and the key of the first key-value message includes the first delivery timestamp, the first delivery number and the message identifier; resubmitting the delivery message to the message middleware according to the first delivery timestamp. Exemplarily, the first key-value message in the embedded database is obtained by calling the Thread.notify method.

[0204] (8) Traverse the embedded database to obtain the first key-value message, and determine whether the first key-value message exists in the embedded database. If the first key-value message does not exist, suspend the re-investment processing thread. If the first key-value message exists, execute (9). Exemplarily, by calling the API seekToFirst method, traverse the instance data (first key-value message) in RocksDB from the beginning.

[0205] (9) Re-deliver the message to the message middleware and determine whether the first key-value message traversed has been delivered. If the re-delivery is completed, continue to traverse the embedded database and perform the re-delivery process.

[0206] (10) Determine whether the message is successfully re-submitted. If successful, execute step (11); if unsuccessful, execute step (12).

[0207] (11) Delete the first key value message corresponding to the message from the embedded database.

[0208] (12) The first delivery number and the message identifier are updated, and the first delivery timestamp is determined according to the updated first delivery number, and the first delivery timestamp, the first delivery number and the message identifier are used as keys and the delivery message is used as a value to write into the embedded database, and the original first key-value message is deleted from the embedded database.

[0209] (13) At the preset cleaning time, start the scheduled task of cleaning data and perform the cleaning of the first key value message. For detailed description, refer to S601 to S603.

[0210] In the disclosed embodiments, the total delivery time of failed messages can be reduced, and in the case where the message middleware failure cannot be recovered in a short time, exponentially delayed delivery is adopted, which will not cause more pressure on the message middleware. Compared with the related technology of using disk resources to replace database resources, it reduces systemic risks and solves the problem of reliable message delivery.

[0211] In the disclosed embodiment, when the message middleware fails and is unavailable as a whole, RocksDB is equivalent to caching all messages, which indirectly solves the problem of message loss due to failure of the message middleware.

[0212] Based on the same inventive concept, the disclosed embodiment also provides a message re-projection device, as described in the following embodiment. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0213] Figure 8 A schematic diagram of a message re-projection device in an embodiment of the present disclosure is shown. Figure 8As shown, the device includes a traversal module 801, a delivery module 802 and a response module 803. The traversal module 801 can be used to traverse the embedded database to obtain a first key-value message; the value of the first key-value message includes a delivery message, and the key of the first key-value message includes a first delivery timestamp and a first delivery number; the delivery message is a message that fails to be delivered to the message middleware; the delivery module 802 can be used to deliver the first key-value message to the message middleware according to the first delivery timestamp; the response module 803 can be used to delete the first key-value message from the embedded database in response to a successful delivery; in response to a failed delivery, update the first delivery number to a second delivery number; determine a second delivery timestamp according to the second delivery number; generate a second key-value message according to the second delivery timestamp, the second delivery number and the delivery message; store the second key-value message as a new first key-value message in the embedded database, and delete the first key-value message that fails to be delivered from the embedded database.

[0214] In one embodiment, the response module 803 may also be configured to determine a delay duration according to the second delivery number; and determine a second delivery timestamp according to the delay duration.

[0215] In one embodiment, the first delivery number and the second delivery number are both exponentially related to the delay time. The response module 803 can also be used to obtain the fault type of the message middleware; determine the delay time according to the fault type and the second delivery number; wherein, when the second delivery number is the same, the longer the fault recovery time of the fault type is, the longer the delay time is determined.

[0216] In one embodiment, the response module 803 can also be used to obtain the average load occupancy of the message middleware within the sliding time window; when the average load occupancy is less than a preset load occupancy threshold, the delay duration is determined based on a preset delay duration initial value, a preset delay duration increment and a second delivery number; and the second delivery timestamp is determined based on the delay duration.

[0217] In one embodiment, the response module 803 may also be used to calculate the second power of the number of delivery times of the delay duration increment to obtain a first value; and calculate the product of the initial value of the delay duration and the first value to obtain the delay duration.

[0218] In one embodiment, after obtaining the average load occupancy rate of the message middleware within the sliding time window, the response module 803 can also be used to obtain the total number of successful deliveries within the sliding time window when the average load occupancy rate is greater than or equal to a preset load occupancy threshold, and obtain the average processing time based on the total number of successful deliveries within the sliding time window and the total time of the sliding time window; determine the delay duration based on a preset delay duration initial value, a preset delay duration increment, the second number of deliveries, the average processing time and the average load occupancy rate; and determine the second delivery timestamp based on the delay duration.

[0219] In one embodiment, the response module 803 can also be used to calculate the second power of the number of deliveries of the delay duration increment to obtain a second value; calculate the product of the initial value of the delay duration and the second value to obtain a third value; calculate the ratio of the average processing time to the total time of the sliding time window to obtain a fourth value; calculate the ratio of the average load occupancy to a preset load occupancy threshold to obtain a fifth value; and determine the delay duration based on the third value, the fourth value and the fifth value.

[0220] In one embodiment, the response module 803 can also be used to obtain a candidate delay duration based on the third value, the fourth value and the fifth value; obtain a preset delay duration threshold; when the candidate delay duration is less than or equal to the delay duration threshold, determine the candidate delay duration as the delay duration; when the candidate delay duration is greater than the delay duration threshold, determine the delay duration threshold as the delay duration.

[0221] In one embodiment, the delivery module 802 can also be used to split the key of the first key-value message to obtain the first delivery timestamp and the first delivery number of the delivery message; when the current time is greater than or equal to the first delivery timestamp and the first delivery number is less than the preset delivery number, the message is re-delivered to the message middleware.

[0222] In the disclosed embodiment, when the message middleware is unavailable due to a failure, a large number of failed delivery messages can be quickly and persistently stored in an embedded database. The embedded database can convert random modifications to the delivery messages into sequential writes, so that tens of millions of messages can be quickly written into the embedded database. It can also be quickly queried from the embedded database, thereby achieving rapid re-delivery of messages and improving the timeliness of the business. By storing failed delivery messages in the embedded database, the database resources of the message middleware are not occupied, and the performance of the database work is guaranteed not to be affected by the failure of message delivery, thereby improving the database performance. In addition, re-delivering messages based on the delivery timestamp determined by the number of delivery times can solve the problem of invalid delivery and reduce the failure rate of message re-delivery.

[0223] It should be noted that the examples and application scenarios implemented by the modules in the above-mentioned device embodiment are the same as those of the corresponding steps in the method embodiment, but are not limited to the contents disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0224] Those skilled in the art will appreciate that various aspects of the present disclosure may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."

[0225] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present disclosure, the electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above message re-transmission methods by executing the executable instructions. Since the principle of solving the problem in the electronic device embodiment is similar to that in the above method embodiment, the implementation of the electronic device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0226] Refer to the following Fig. 9 The electronic device 900 according to this embodiment of the present disclosure is described. Fig. 9 The electronic device 900 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0227] like Fig. 9 As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include but are not limited to: at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910).

[0228] The storage unit stores program codes, which can be executed by the processing unit 910, so that the processing unit 910 executes the steps described in the above “Exemplary Method” section of this specification according to various exemplary embodiments of the present disclosure.

[0229] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 9201 and / or a cache storage unit 9202 , and may further include a read-only storage unit (ROM) 9203 .

[0230] The storage unit 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0231] Bus 930 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0232] The electronic device 900 may also communicate with one or more external devices 940 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 900, and / or may communicate with any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 950. Furthermore, the electronic device 900 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 960. As shown, the network adapter 960 communicates with other modules of the electronic device 900 via a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0233] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0234] In the disclosed exemplary embodiments, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. Fig.10 A schematic diagram of a computer-readable storage medium in an embodiment of the present disclosure is shown. Fig.10 As shown, the computer-readable storage medium 1000 stores a program product capable of implementing the above method of the present disclosure.

[0235] In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary implementations of the present disclosure described in the above "Specific Implementation Methods" section of this specification.

[0236] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0237] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0238] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0239] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0240] Based on the same inventive concept, a computer program product is also provided in an embodiment of the present disclosure, including a computer program product, including: a computer program or an instruction, wherein when the computer program or the instruction is executed by a processor, the message retransmission method of any one of the above method embodiments is implemented. Since the principle of solving the problem in the computer program product embodiment is similar to that in the above method embodiment, the implementation of the computer program product embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0241] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0242] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0243] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A message re-delivery method, characterized in that: include: Traversing the embedded database to obtain a first key-value message, wherein the value of the first key-value message includes a delivery message, the key of the first key-value message includes a first delivery timestamp and a first delivery number, and the delivery message is a message that fails to be delivered to the message middleware; Delivering the first key-value message to the message middleware according to the first delivery timestamp; In response to successful delivery, deleting the first key-value message from the embedded database; In response to a delivery failure, updating the first delivery number to a second delivery number; Determining a second delivery timestamp according to the second delivery number; Generate a second key-value message according to the second delivery timestamp, the second delivery number and the delivery message; store the second key-value message as a new first key-value message in the embedded database, and delete the first key-value message that fails to be delivered from the embedded database.

2. The method according to claim 1, characterized in that Determining a second delivery timestamp according to the second delivery number includes: Determine the delay time according to the second delivery number; A second delivery timestamp is determined according to the delay duration.

3. The method according to claim 2, characterized in that The first delivery number and the second delivery number are both exponentially related to the delay duration.

4. The method according to claim 2, characterized in that: Determining the delay time according to the second delivery number includes: Obtaining the fault type of the message middleware; Determine a delay duration according to the fault type and the second delivery number; Among them, when the second delivery times are the same, the longer the fault recovery time of the fault type is, the longer the delay time is determined.

5. The method according to claim 1, characterized in that Determining a second delivery timestamp according to the second delivery number includes: Obtaining the average load occupancy rate of the message middleware within the sliding time window; When the average load occupancy rate is less than a preset load occupancy threshold, determining the delay duration according to a preset delay duration initial value, a preset delay duration increment and the second delivery number; A second delivery timestamp is determined according to the delay duration.

6. The method according to claim 5, characterized in that Determining the delay duration according to a preset delay duration initial value, a preset delay duration increment, and the second delivery number includes: Calculate the delay time increment raised to the power of the second number of deliveries to obtain a first value; The product of the initial value of the delay duration and the first value is calculated to obtain the delay duration.

7. The method according to claim 5, characterized in that After obtaining the average load occupancy rate of the message middleware within the sliding time window, the method further includes: When the average load occupancy rate is greater than or equal to the preset load occupancy threshold, the total number of successful deliveries within the sliding time window is obtained, and the average processing time is obtained according to the total number of successful deliveries within the sliding time window and the total time of the sliding time window; the delay duration is determined according to a preset delay duration initial value, a preset delay duration increment, the second number of deliveries, the average processing time and the average load occupancy rate; A second delivery timestamp is determined according to the delay duration.

8. The method according to claim 7, characterized in that Determining the delay duration according to a preset delay duration initial value, a preset delay duration increment, the second delivery number, the average processing time, and the average load occupancy rate includes: Calculate the delay time increment raised to the power of the second number of deliveries to obtain a second value; Calculate the product of the initial value of the delay time and the second value to obtain a third value; Calculating a ratio of the average processing time to the total time of the sliding time window to obtain a fourth value; Calculating a ratio of the average load occupancy rate to the preset load occupancy threshold to obtain a fifth value; A delay duration is determined based on the third value, the fourth value, and the fifth value.

9. The method according to claim 8, characterized in that Determining a delay duration based on the third value, the fourth value, and the fifth value includes: Obtaining a candidate delay duration based on the third value, the fourth value, and the fifth value; Get the preset delay time threshold; When the candidate delay duration is less than or equal to the delay duration threshold, determining the candidate delay duration as the delay duration; In a case where the candidate delay duration is greater than the delay duration threshold, the delay duration threshold is determined as the delay duration.

10. The method according to claim 1, characterized in that Delivering the first key-value message to the message middleware according to the first delivery timestamp includes: Splitting the key of the first key-value message to obtain a first delivery timestamp and a first delivery number of the delivery message; When the current time is greater than or equal to the first delivery timestamp and the first delivery number is less than the preset delivery number, the delivery message is delivered to the message middleware.

11. A message re-posting device, characterized in that: include: A traversal module, used for traversing the embedded database to obtain a first key-value message; the value of the first key-value message includes a delivery message, the key of the first key-value message includes a first delivery timestamp and a first delivery number; the delivery message is a message that fails to be delivered to the message middleware; A delivery module, configured to deliver the first key-value message to the message middleware according to the first delivery timestamp; A response module, configured to delete the first key-value message from the embedded database in response to successful delivery; In response to a delivery failure, updating the first delivery number to a second delivery number; determining a second delivery timestamp according to the second delivery number; Generate a second key-value message according to the second delivery timestamp, the second delivery number and the delivery message; store the second key-value message as a new first key-value message in the embedded database, and delete the first key-value message that fails to be delivered from the embedded database.

12. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the message re-projection method described in any one of claims 1 to 10 by executing the executable instructions.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the message re-direction method described in any one of claims 1 to 10 is implemented.

14. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed by a processor, it implements the message re-delivery method described in any one of claims 1 to 10.

Images