A service persistence method and device

Abstract

This application provides a business persistence method and apparatus. The method includes: an online processing device generating a corresponding business message based on fields to be stored in a database, information representing the category, and identifier information uniquely identifying the request information in a request message; the online processing device sending the business message and message queue information corresponding to the category of the business message to a message queue device; the message queue device allocating the business message to a message queue of the corresponding category based on the message queue information corresponding to the category of the business message; a database storage processing device obtaining multiple business messages of each category from multiple message queues of multiple categories; wherein, each category's message queue caches business messages of the corresponding category; and the database storage processing device performing batch persistence processing on multiple business messages of any category obtained, based on the persistence method corresponding to the category.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of financial technology (Fintech), and in particular to a business persistence method and device. BACKGROUND

[0002] With the development of computer technology, more and more technologies are applied in the financial field, and the traditional financial industry is gradually changing to financial technology. However, due to the security and real-time requirements of the financial industry, higher requirements are also put forward for technology. In the financial field, it is necessary to perform a persistence operation on a business, that is, to convert transient data into persistent data.

[0003] In the related art, after processing data of each business, the business is directly subjected to a persistence operation. However, the above scheme has low efficiency for performing a persistence operation on a single business.

[0004] In summary, there is an urgent need for a business persistence method to improve the efficiency of the persistence operation. SUMMARY

[0005] Embodiments of the present application provide a business persistence method and device to improve the efficiency of the persistence operation.

[0006] In a first aspect, the embodiments of the present application provide a business persistence method applied to a persistence system, wherein the persistence system includes an online processing device, a message queue device, and a storage processing device; and the method includes:

[0007] The online processing device generates a corresponding business message based on a field needing to be stored in the request information, information representing a category, and identification information uniquely identifying the request information;

[0008] The online processing device sends the business message and message queue information corresponding to the category of the business message to the message queue device;

[0009] The message queue device distributes the business message to a message queue of a corresponding category based on the message queue information corresponding to the category of the business message;

[0010] The storage processing device respectively acquires a plurality of business messages of each category from a plurality of message queues of different categories, wherein the message queue of each category caches business messages of the corresponding category;

[0011] The storage processing device performs batch persistence processing on the plurality of business messages of any category based on a persistence mode corresponding to the category.

[0012] The above scheme, in the category dimension, the message queue device distributes the service messages sent by the online processing device to the message queue of the corresponding category, and then the warehousing processing device obtains multiple service messages of each category, and uses the persistent mode corresponding to the category to perform persistent processing on the multiple service messages of the category, thereby parallelly processing the service messages of multiple categories, improving the persistent operation efficiency and accuracy; and by buffering the service messages of the corresponding category through the message queue, and then batch processing the multiple service messages of each category for persistent processing, the data persistence and the processing of the business logic are decoupled, the persistent response speed is improved, and the persistent operation efficiency is further improved.

[0013] In some optional embodiments, the batch persistent processing of the multiple service messages of the category based on the persistent mode corresponding to the category comprises:

[0014] The write pointer of the buffer corresponding to the category is used to sequentially write the service messages of the category into the buffer positions of the buffer in the order of the buffer positions of the buffer, and the read pointer of the buffer is used to sequentially read the service messages written by the write pointer in the order of the buffer positions of the buffer, and store the fields needing to be warehoused in the read service messages into the storage position corresponding to the category.

[0015] The above scheme sequentially writes the service messages of any category into the buffer positions of the buffer in the order of the buffer positions of the buffer corresponding to the category through the write pointer, and sequentially reads the service messages through the read pointer, thereby realizing fast reading of the service messages, and then storing the fields needing to be warehoused in the read service messages into the storage position corresponding to the category, avoiding the influence of the persistent efficiency caused by the problems such as deadlock.

[0016] In some optional embodiments, the method further comprises:

[0017] If the write pointer is adjacent to the read pointer and is after the read pointer, the warehousing processing device suspends writing the service messages of the category into the buffer through the write pointer, and continues to write the service messages of the category into the buffer through the write pointer after the write pointer is not adjacent to the read pointer; or

[0018] If the read pointer is adjacent to the write pointer and is after the write pointer, the warehousing processing device suspends reading the service messages through the read pointer, and continues to read the service messages through the read pointer after the read pointer is not adjacent to the write pointer.

[0019] If the write pointer is adjacent to the read pointer and after the read pointer, it indicates that the existing service message written in the buffer position through the write pointer has not been read, and the write of the service message of the category through the write pointer is suspended, i.e., the existing service message is not overwritten by the service message to be written, so as to avoid the loss of the existing service message. After the write pointer is not adjacent to the read pointer, it indicates that the existing service message is read through the read pointer, and the write of the service message of the category through the write pointer is enabled, i.e., the existing service message read is overwritten by the service message to be written. If the read pointer is adjacent to the write pointer and after the write pointer, it indicates that all the written service messages are read through the read pointer, and the read of the service message through the read pointer is suspended. After the read pointer is not adjacent to the write pointer, it indicates that the service message is written through the write pointer, and the read of the new written service message through the read pointer is enabled.

[0020] In some optional embodiments, the writing of the service message of the category into the buffer position of the buffer in sequence comprises:

[0021] If the buffer position to be written has the service message, the service message to be written is used to overwrite the existing service message.

[0022] In the above solution, when the buffer position to be written has the service message, the service message to be written is used to overwrite the existing service message, instead of deleting the existing service message and then writing the service message to be written into the buffer position, so that the time-consuming of recycling the useless memory space is saved, and the efficiency of the persistent operation is further improved.

[0023] In some optional embodiments, before the batch persistent processing of the plurality of service messages of the category based on the persistent mode corresponding to the category, the method further comprises:

[0024] The storage processing device saves the identification information of the plurality of service messages of the category in a remote dictionary service (Redis) for a preset time length, and performs the deduplication processing on the plurality of service messages of the category based on a data deduplication algorithm of the Redis.

[0025] In the above solution, due to network fluctuations and other factors, the service message may be repeated in a short time. The data deduplication algorithm of the Redis is used to perform the deduplication processing on the service message in the preset time length, so as to avoid the repeated persistent of the same service message, and reduce the situation that the Redis memory is occupied due to the long time of saving the identification information of the service message.

[0026] In some optional embodiments, for any category of message queue, at least one sub-queue is included;

[0027] The number of sub-queues of any category is adjusted in the following manner:

[0028] If the ratio between the total number of service messages cached by all sub-queues of the category and the number of sub-queues of the category is greater than a first ratio, the number of sub-queues of the category is increased; or

[0029] If the ratio between the total number of service messages cached by all sub-queues of the category and the number of sub-queues of the category is less than a second ratio, and the number of sub-queues of the category is greater than a preset number, the number of sub-queues of the category is decreased.

[0030] The first ratio is greater than the second ratio.

[0031] In the above scheme, if the ratio between the total number of service messages cached by all sub-queues of a category and the number of sub-queues of the category is greater than the larger first ratio, it indicates that the total number of service messages cached by all sub-queues of the category is relatively large, and the number of sub-queues of the category cannot meet the caching requirement. Based on this, by increasing the number of sub-queues of the category, more sub-queues are used to cache service messages of the category, so that the number of sub-queues of the category meets the caching requirement. If the ratio between the total number of service messages cached by all sub-queues of a category and the number of sub-queues of the category is less than the smaller first ratio, it indicates that the total number of service messages cached by all sub-queues of the category is relatively small, and the sub-queues are relatively idle. After it is determined that the number of sub-queues of the category is greater than a preset number, the number of sub-queues of the category can be decreased.

[0032] In some optional embodiments, the persistent system further includes a load balancer, and there are multiple online processing devices in the persistent system; before the online processing device generates a corresponding service message based on the field of the request information that needs to be warehoused, the information representing the category, and the identification information uniquely identifying the request information, the method includes:

[0033] The load balancer sequentially selects an online processing device from all online processing devices based on a sequential circular queue, and sends to-be-processed information to the selected online processing device; wherein the to-be-processed information includes the request information, the information representing the category, and the identification information uniquely identifying the request information.

[0034] In some optional embodiments, the method further includes:

[0035] If an online processing device fails, the load balancer removes the failed online processing device from the sequential circular queue.

[0036] In a second aspect, the embodiments of the present application provide a service persistence method, applied to a warehousing processing device, the method comprising:

[0037] acquiring a plurality of service messages of each category from a plurality of category message queues, wherein each category message queue stores service messages of the corresponding category;

[0038] performing batch persistence processing on the plurality of service messages of the category based on the persistence mode corresponding to the category.

[0039] In some optional embodiments, performing batch persistence processing on the plurality of service messages of the category based on the persistence mode corresponding to the category comprises:

[0040] writing the service messages of the category into the buffer positions of the buffer in sequence according to the buffer position order of the buffer through the write pointer of the buffer corresponding to the category; and reading the service messages written through the write pointer in sequence according to the buffer position order of the buffer through the read pointer of the buffer, and storing the fields needing to be warehoused in the read service messages into the storage position corresponding to the category.

[0041] In some optional embodiments, the method further comprises:

[0042] if the write pointer is adjacent to the read pointer and is behind the read pointer, suspending the writing of the service messages of the category into the buffer through the write pointer, and continuing the writing of the service messages of the category into the buffer through the write pointer after the write pointer is not adjacent to the read pointer; or

[0043] if the read pointer is adjacent to the write pointer and is behind the write pointer, suspending the reading of the service messages through the read pointer, and continuing the reading of the service messages through the read pointer after the read pointer is not adjacent to the write pointer.

[0044] In some optional embodiments, writing the service messages of the category into the buffer positions of the buffer in sequence comprises:

[0045] if the buffer position to be written has service messages, the service messages needing to be written are used to overwrite the existing service messages.

[0046] In some optional embodiments, before performing batch persistence processing on the plurality of service messages of the category based on the persistence mode corresponding to the category, the method further comprises:

[0047] The identification information of the plurality of service messages of the category is stored in Redis for a preset time length, and the plurality of service messages of the category are processed by a data deduplication algorithm based on Redis.

[0048] In some optional embodiments, for any category of message queue, at least one subqueue is included;

[0049] The number of subqueues of any category is adjusted in the following manner:

[0050] If the ratio between the total number of service messages cached by all subqueues of the category and the number of subqueues of the category is greater than a first ratio, the number of subqueues of the category is increased; or

[0051] If the ratio between the total number of service messages cached by all subqueues of the category and the number of subqueues of the category is less than a second ratio, and the number of subqueues of the category is greater than a preset number, the number of subqueues of the category is decreased.

[0052] The first ratio is greater than the second ratio.

[0053] In a third aspect, an embodiment of the present application further provides a service persistence device, comprising:

[0054] A service message acquisition module is configured to acquire a plurality of service messages of each category from a plurality of categories of message queues respectively; wherein each category of message queue caches service messages of the corresponding category;

[0055] A persistence module is configured to perform batch persistence processing on the plurality of service messages of any category based on the persistence mode corresponding to the category.

[0056] In a fourth aspect, an embodiment of the present application further provides a persistence system, comprising an online processing device, a message queue device, and an ingression processing device; wherein:

[0057] The online processing device is configured to generate a corresponding service message based on a field needing ingression in request information, information representing a category, and identification information uniquely identifying the request information.

[0058] The online processing device is further configured to send the service message and message queue information corresponding to the category of the service message to the message queue device.

[0059] The message queue device is configured to distribute the service message to a message queue of the corresponding category based on the message queue information corresponding to the category of the service message.

[0060] The warehousing processing device is configured to obtain a plurality of service messages of each category from a plurality of category message queues respectively, wherein the category message queues store service messages of the corresponding category;

[0061] The warehousing processing device is further configured to perform batch persistence processing on the plurality of service messages of any category based on the persistence mode corresponding to the category.

[0062] In a fifth aspect, an embodiment of the present application provides a warehousing processing device, comprising at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor, the processor executes the service persistence method of any one of the second aspect.

[0063] In a sixth aspect, an embodiment of the present application provides a computer readable storage medium storing a computer program executable by a computing device, and when the program runs on the computing device, the computing device executes the service persistence method of any one of the second aspect.

[0064] In addition, the technical effects brought by any one of the implementation manners of the second aspect to the sixth aspect can refer to the technical effects brought by different implementation manners of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0066] Figure 1 The first kind of persistence system architecture diagram provided by the embodiment of the present application;

[0067] Figure 2 The interaction flowchart of the first kind of service persistence method provided by the embodiment of the present application;

[0068] Figure 3 The first kind of message queue schematic diagram provided by the embodiment of the present application;

[0069] Figure 4 The second kind of message queue schematic diagram provided by the embodiment of the present application;

[0070] Figure 5 The third kind of message queue schematic diagram provided by the embodiment of the present application;

[0071] Figure 6A second persistent system architecture diagram provided by an embodiment of the present application;

[0072] Figure 7 An interaction flowchart of a second service persistence method provided by an embodiment of the present application;

[0073] Figure 8 A buffer diagram provided by an embodiment of the present application;

[0074] Figure 9 A first read pointer and write pointer position diagram provided by an embodiment of the present application;

[0075] Figure 10 A second read pointer and write pointer position diagram provided by an embodiment of the present application;

[0076] Figure 11 An interaction flowchart of a third service persistence method provided by an embodiment of the present application;

[0077] Figure 12 A third persistent system architecture diagram provided by an embodiment of the present application;

[0078] Figure 13 An interaction flowchart of a fourth service persistence method provided by an embodiment of the present application;

[0079] Figure 14 A flowchart of a service persistence method provided by an embodiment of the present application;

[0080] Figure 15 A structure diagram of a service persistence device provided by an embodiment of the present application;

[0081] Figure 16 A structure diagram of a warehouse-in processing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0082] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0083] In the financial field, it is necessary to perform a persistence operation on a service, that is, to convert transient data into persistent data.

[0084] In some embodiments, after data processing is performed on each service message, a persistence operation is directly performed on the service message, that is, part of the fields in the service message are stored into a database.

[0085] However, in the scenario of generating a large number of business messages in a short time, a large amount of time is spent on persisting each business, resulting in low efficiency of the entire processing flow, and the persistence logic irrelevant to business processing needs to be coupled in each business.

[0086] In view of this, the embodiments of the present application provide a business persistence method and device, the method is applied to a warehousing processing device, comprising: obtaining a plurality of business messages of each category from a plurality of category message queues; wherein the business messages of the corresponding category are cached in the message queue of each category; for the plurality of business messages of any category obtained, performing batch persistence processing on the plurality of business messages of the category based on the persistence mode corresponding to the category. The method obtains a plurality of business messages of each category in the category dimension, and uses the persistence mode corresponding to the category to perform persistence processing on the plurality of business messages of the category, thereby performing parallel persistence processing on the business messages of multiple categories, improving the persistence operation efficiency and accuracy; and by caching the business messages of the corresponding category through the message queue, and then performing batch persistence processing on the plurality of business messages of each category, the data persistence and the processing of the business logic are decoupled, the persistence response speed is improved, and the persistence operation efficiency is further improved.

[0087] Referring to Figure 1 The first persistence system architecture diagram provided by the embodiments of the present application is shown, and the persistence system 10 comprises an online processing device 100, a message queue device 200, and a warehousing processing device 300.

[0088] The online processing device 100 is configured to generate a corresponding business message based on the fields needing warehousing in the request information, the information representing the category, and the identification information uniquely identifying the request information.

[0089] The online processing device 100 is further configured to send the business message and the message queue information corresponding to the category of the business message to the message queue device 200.

[0090] The message queue device 200 is configured to distribute the business message to the message queue of the corresponding category based on the message queue information corresponding to the category of the business message, that is, the message queue device 200 is provided with a plurality of category message queues, and the business messages of the corresponding category are cached in the message queue of each category.

[0091] The warehousing processing device 300 is configured to obtain a plurality of business messages of each category from a plurality of category message queues; and for the plurality of business messages of any category obtained, performing batch persistence processing on the plurality of business messages of the category based on the persistence mode corresponding to the category.

[0092] The above-mentioned persistence system is only illustrative, and the embodiments of the present application are not limited to the above-mentioned persistence system.

[0093] The technical solutions of the present application and how the technical solutions of the present application solve the above-mentioned technical problems will be described in detail below in combination with the drawings and specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments.

[0094] Figure 2 The first service persistence method provided by the embodiments of the present application has a flow chart of interaction as shown in Figure 2 , which includes the following steps:

[0095] Step S201: The online processing device generates a corresponding service message based on the fields needing to be stored in the request information, the information representing the category, and the identification information uniquely identifying the request information.

[0096] The above-mentioned service message is generated by the online processing device based on the fields needing to be stored in the request information, the information representing the category, and the identification information uniquely identifying the request information.

[0097] In implementation, the fields needing to be stored in different categories of request information are different, for example, the fields of serial number, customer number, customer name, transaction time, corresponding information, and error information in the request information of category A are the fields needing to be stored, and the fields of serial number, customer number, customer name, transaction time, corresponding code, and error information in the request information of category B are the fields needing to be stored. The fields needing to be stored in the request information of different categories are only illustrative, and the present application does not make specific limitations thereto.

[0098] In some optional implementation, an annotation can be added to the fields needing to be stored in the request information, which represents that the corresponding field needs to be stored, so as to facilitate the online processing device to determine the fields needing to be stored in the request information based on the annotation.

[0099] The above-mentioned identification information can uniquely identify the request information. Since the serial number generated by the self-increment method may be repeated in a distributed scenario, the identification information can be generated based on part of the information in the request information, for example, the channel number, system number, machine room id, work machine id, timestamp, and serial number in the request information are combined as the above-mentioned identification information.

[0100] The online processing device can generate the corresponding service message by but not limited to the following ways:

[0101] Based on the above annotation, determine the fields in the request information that need to be stored in the database, and generate a json format (a lightweight data exchange format) business message corresponding to the feature values, feature values of the characteristic category, and feature values of the characteristic identifier of these fields that need to be stored in the database.

[0102] Step S202: The online processing device sends the business message and the message queue information corresponding to the category of the business message to the message queue device.

[0103] The online processing device can send the business message and the message queue information corresponding to the category of the business message to the message queue device in the following ways, but is not limited to:

[0104] The above-mentioned json format business message and the message queue information corresponding to the category of the business message are sent to the message queue device as the message body of the message queue.

[0105] Step S203: The message queue device distributes the business message to the message queue corresponding to the category.

[0106] Referring to Figure 3 , a message queue diagram provided by the present embodiment is shown, the message queue device is configured with five categories of message queues, wherein the message queue of category A is used to cache business messages of category A; the message queue of category B is used to cache business messages of category B; the message queue of category C is used to cache business messages of category C; the message queue of category D is used to cache business messages of category D; and the message queue of category E is used to cache business messages of category E.

[0107] In the implementation, the business messages cached in the message queues of each category are all saved in the message storage file (commit log), so as to avoid the loss of business messages.

[0108] Figure 3 It is only an exemplary description of the message queue, and more or less message queues can be set in actual application.

[0109] Step S204: The storage processing device acquires a plurality of business messages of each category from the message queues of a plurality of categories.

[0110] For example, by inheriting the business message listener of the message queue in the storage processing device, a plurality of business messages of each category are acquired.

[0111] The storage processing device can acquire a plurality of business messages of each category in the following ways, but is not limited to:

[0112] 1) The storage processing device receives a plurality of business messages pushed by the message queue of each category;

[0113] 2) The warehousing processing device pulls multiple service messages from the message queue of each category.

[0114] The above two acquisition methods are only illustrative, and the application is not limited thereto.

[0115] Step S205: For any category of multiple service messages obtained, the warehousing processing device performs batch persistence processing on the multiple service messages of the category based on the persistence mode corresponding to the category.

[0116] In this embodiment, the persistence processing is performed in the category dimension, thereby parallelly processing the service messages of multiple categories. Moreover, the batch persistence processing of the multiple service messages of each category improves the persistence operation efficiency.

[0117] Still taking the categories A-E as an example, the following steps can be executed in parallel:

[0118] Based on the persistence mode corresponding to category A, the obtained multiple service messages of category A are processed for persistence;

[0119] Based on the persistence mode corresponding to category B, the obtained multiple service messages of category B are processed for persistence;

[0120] Based on the persistence mode corresponding to category C, the obtained multiple service messages of category C are processed for persistence;

[0121] Based on the persistence mode corresponding to category D, the obtained multiple service messages of category D are processed for persistence;

[0122] Based on the persistence mode corresponding to category E, the obtained multiple service messages of category E are processed for persistence.

[0123] The above persistence processing process is only illustrative, and the application is not limited thereto.

[0124] In the above technical solution, the message queue device distributes the service messages sent by the online processing device to the message queue of the corresponding category, and then the warehousing processing device obtains multiple service messages of each category and processes the multiple service messages of the category using the persistence mode corresponding to the category, thereby parallelly processing the service messages of multiple categories, improving the persistence operation efficiency and accuracy. Moreover, the message queue buffers the service messages of the corresponding category, and then performs batch persistence processing on the multiple service messages of each category, decouples the data persistence from the processing of the business logic, improves the persistence response speed, and further improves the persistence operation efficiency.

[0125] In actual application, the number of service messages of each category generated in a period of time is different, and thus the number of service messages cached by the message queue of different categories is different. Based on this, Figure 4 Another message queue schematic diagram provided for the embodiment is shown in FIG. 2. For any category of message queue, at least one subqueue is included. The number of subqueues of a certain category is adjusted according to the total number of service messages cached by all subqueues of the category. Referring to FIG. 2, Figure 4 As shown in FIG. 2, the message queue of category A includes subqueue A1 and subqueue A2; the message queue of category B includes subqueue B1, subqueue B2 and subqueue B3; the message queue of category C includes subqueue C1; the message queue of category D includes subqueue D1 and subqueue D2; and the message queue of category E includes subqueue E1, subqueue E2 and subqueue E3.

[0126] In some optional embodiments, the number of subqueues of any category is adjusted in the following manner:

[0127] If the ratio between the total number of service messages cached by all subqueues of the category and the number of subqueues of the category is greater than a first ratio, the number of subqueues of the category is increased; or

[0128] If the ratio between the total number of service messages cached by all subqueues of the category and the number of subqueues of the category is less than a second ratio, and the number of subqueues of the category is greater than a preset number, the number of subqueues of the category is decreased.

[0129] The first ratio is greater than the second ratio.

[0130] The first ratio is greater than the second ratio, i.e., the first ratio is a larger ratio compared with the second ratio.

[0131] If the ratio between the total number of service messages cached by all subqueues of a certain category and the number of subqueues of the category is greater than the larger first ratio, it indicates that the total number of service messages cached by all subqueues of the category is relatively large, and the number of subqueues of the category cannot meet the caching requirement. Based on this, the number of subqueues of the category is increased, so that more subqueues can cache service messages of the category, thereby meeting the caching requirement of the number of subqueues of the category.

[0132] Conversely, if the ratio between the total number of service messages cached by all subqueues of a certain category and the number of subqueues of the category is less than the smaller first ratio, it indicates that the total number of service messages cached by all subqueues of the category is relatively small, and the subqueues are relatively idle. After it is determined that the number of subqueues of the category is greater than a preset number, the number of subqueues of the category can be decreased.

[0133] The above-mentioned decrease of the number of subqueues of a certain category can be implemented in the following manner, but is not limited thereto:

[0134] 1) randomly select a sub-queue from all sub-queues of the category, allocate the buffered service messages in the selected sub-queue to other sub-queues of the category, and then delete the selected sub-queue;

[0135] 2) determine a sub-queue with the least buffered service messages from all sub-queues of the category, allocate the buffered service messages in the sub-queue with the least buffered service messages to other sub-queues of the category, and then delete the sub-queue with the least buffered service messages.

[0136] The above two ways of reducing the number of sub-queues of a category are only illustrative, and the present application is not limited thereto.

[0137] The above preset number can be set according to actual application scenarios. In some embodiments, the preset number can be 1, that is, the number of sub-queues of a category cannot be reduced to 0, so as to ensure that each category has at least one sub-queue.

[0138] Still taking Figure 4 as an example, the first ratio is denoted as R0, the second ratio is denoted as R0', and the preset number is 1. The number of sub-queues is adjusted in the following way:

[0139] The ratio between the total number of buffered service messages of all sub-queues of category A and the number of sub-queues of category A is denoted as R1, R1>R0, and sub-queue A3 of category A is increased.

[0140] The ratio between the total number of buffered service messages of all sub-queues of category B and the number of sub-queues of category B is denoted as R2, R2

[0141] The ratio between the total number of buffered service messages of all sub-queues of category C and the number of sub-queues of category C is denoted as R3, R3

[0142] The ratio between the total number of buffered service messages of all sub-queues of category D and the number of sub-queues of category D is denoted as R4, R4 neither greater than R0 nor less than R0', and the number of sub-queues of category D does not need to be adjusted.

[0143] The ratio between the total number of buffered service messages of all sub-queues of category E and the number of sub-queues of category E is denoted as R5, and similarly, R5 neither greater than R0 nor less than R0', and the number of sub-queues of category E does not need to be adjusted.

[0144] The message queue after the adjustment of the sub-queues can refer to Figure 5 As shown in the figure, the message queue of category A includes sub-queue A1, sub-queue A2 and sub-queue A3; the message queue of category B includes sub-queue B1 and sub-queue B2; the message queue of category C includes sub-queue C1; the message queue of category D includes sub-queue D1 and sub-queue D2; and the message queue of category E includes sub-queue E1, sub-queue E2 and sub-queue E3.

[0145] The above example is only one possible way to adjust the number of sub-queues, and the present application is not limited thereto.

[0146] Referring to Figure 6 As shown in the figure, the second persistent system architecture diagram provided by the embodiment, on the basis of the persistent system 10 shown in the figure, Figure 1 The persistent system 20 further includes a monitoring device 400, and the adjustment of the number of sub-queues of any category can be performed by the monitoring device 400.

[0147] In addition, the monitoring device can also compare the first number of business messages obtained by the warehousing processing device in a unit time and the second number of business messages sent by the online processing device to the message queue device, and give a warning according to the comparison result.

[0148] For example, if the first number is less than the second number, and the deviation of the second number compared with the first number is greater than a preset deviation, it means that the speed of the online processing device sending business messages to the message queue device is much greater than the speed of the warehousing processing device obtaining business messages, which will cause the commitlog storage space in the message queue device to be insufficient. Therefore, relevant operation and maintenance personnel can be notified in a timely manner by means of email, short message, telephone and the like to monitor and adjust the processing capacity of the warehousing processing device.

[0149] Figure 7 The second business persistence method provided by the embodiment of the present application has the interaction flowchart as shown in the figure. Figure 7 As shown in the figure, the second business persistence method provided by the embodiment of the present application has the interaction flowchart as shown in the figure.

[0150] Step S701: The online processing device generates a corresponding business message based on the fields needing warehousing in the request information, the information representing the category and the identification information uniquely identifying the request information.

[0151] Step S702: The online processing device sends the business message and the message queue information corresponding to the category of the business message to the message queue device.

[0152] Step S703: The message queue device distributes the business message to the message queue of the corresponding category.

[0153] Step S704: The inbound processing device retrieves multiple business messages of each category from multiple message queues.

[0154] The specific implementation of steps S701 to S704 can be found in the above embodiments, and will not be repeated here.

[0155] Step S705: For multiple business messages of any category, the data entry processing device writes the business messages of the category sequentially to the buffer positions of the buffer according to the buffer position order, using the write pointer of the buffer corresponding to the category; and reads the business messages written by the write pointer sequentially according to the buffer position order, using the read pointer of the buffer, and stores the fields of the read business messages that need to be entered into the data entry into the storage position corresponding to the category.

[0156] This embodiment requires batch persistence of multiple business messages of a single category. If a queue within a concurrent packet is used to collect a certain number of business messages before storing them in the corresponding storage location, a large number of messages of that category may not be stored in time, potentially leading to memory shortages. If a locked buffered queue is used for persistence, deadlocks may occur, impacting persistence efficiency.

[0157] Based on this, in this embodiment, for any category, the business messages of that category are sequentially written to the buffer positions of the corresponding buffer using a write pointer, and sequentially read using a read pointer. This enables fast reading of business messages, and then the fields in the read business messages that need to be stored in the database are stored in the storage location corresponding to that category.

[0158] See Figure 8 As shown, there are 8 buffer positions corresponding to category A, which are denoted as buffer position 1, buffer position 2, buffer position 3, buffer position 4, buffer position 5, buffer position 6, buffer position 7 and buffer position 8 in order. The 100 business messages of category A (denoted as business message 1, business message 2, business message 3, ... business message 99, business message 100) are written to buffer positions 1 to 8 respectively by writing pointers.

[0159] Figure 8 This is an exemplary illustration of a buffer. In practical applications, the buffer may have more or fewer buffer locations, and this application does not make any specific limitations on this.

[0160] In some embodiments, for performance reasons, the number of buffer locations can be set to 2. n .

[0161] In implementation, the number of service messages of a certain category can be greater than the number of buffer positions. For example, in the above embodiment, there are 100 service messages of category A, and the buffer area has only 8 buffer positions. Obviously, it is impossible to write all the service messages into the buffer positions of the buffer area only once.

[0162] Based on this, if the buffer position to be written has no service message, the service message to be written is directly written into the buffer position.

[0163] If the buffer position to be written has a service message, the service message to be written is written to overwrite the existing service message.

[0164] By overwriting the existing service message with the service message to be written when the buffer position to be written has a service message, instead of deleting the existing service message and then writing the service message to be written into the buffer position, the time-consuming of recycling the useless memory space is saved, and the efficiency of the persistent operation is further improved.

[0165] Still taking the above Figure 8 For example, service message 1 is directly written into buffer position 1; service message 2 is written into buffer position 2; service message 3 is written into buffer position 3; service message 4 is written into buffer position 4; service message 5 is written into buffer position 5; service message 6 is written into buffer position 6; service message 7 is written into buffer position 7; and service message 8 is written into buffer position 8.

[0166] Service message 9 is written to overwrite service message 1 in buffer position 1; service message 10 is written to overwrite service message 2 in buffer position 2; service message 11 is written to overwrite service message 3 in buffer position 3; service message 12 is written to overwrite service message 4 in buffer position 4; service message 13 is written to overwrite service message 5 in buffer position 5; service message 14 is written to overwrite service message 6 in buffer position 6; service message 15 is written to overwrite service message 7 in buffer position 7; and service message 16 is written to overwrite service message 8 in buffer position 8.

[0167] Service message 97 is written to overwrite service message 89 in buffer position 1; service message 98 is written to overwrite service message 90 in buffer position 2; service message 99 is written to overwrite service message 91 in buffer position 3; and service message 100 is written to overwrite service message 92 in buffer position 4.

[0168] The above example is a feasible implementation manner of writing service messages, and the present application is not limited thereto.

[0169] As mentioned above, the number of business messages of a certain category may exceed the number of buffer locations. When there are business messages in a buffer location to be written, the business message to be written will overwrite the existing business messages. If an existing business message is overwritten by the business message to be written before it has been read, the existing business message will be lost and cannot be stored in the corresponding storage location, resulting in persistence failure. Therefore, before overwriting an existing business message, it is necessary to ensure that the existing business message has been read.

[0170] In some optional implementations, if the write pointer is adjacent to the read pointer and follows the read pointer, it means that the existing business message written to the buffer position through the write pointer has not yet been read. At this time, the writing of the business message of this type to the buffer through the write pointer is paused, that is, the business message to be written will no longer overwrite the existing business message, thus avoiding the loss of the existing business message.

[0171] Furthermore, when the write pointer and read pointer are no longer adjacent, it means that an existing business message has been read again through the read pointer. At this time, the business message of this type can be written to the buffer through the write pointer, that is, the business message to be written will continue to overwrite the existing business message that has been read.

[0172] In some alternative implementations, if the read pointer is adjacent to the write pointer and follows the write pointer, it means that all written business messages have been read through the read pointer. At this point, there are no more written business messages to read, so it is necessary to pause reading business messages through the read pointer.

[0173] Furthermore, if the read pointer and write pointer are no longer adjacent, it means that a business message has been written again through the write pointer. At this point, the newly written business message can be read again through the read pointer.

[0174] See Figure 9 As shown, the write pointer is at buffer position 4 and the read pointer is at buffer position 5. The business messages written through the write pointer to buffer positions 1 to 8 have not been read. At this time, the writing of business messages of this type to the buffer is paused. After the read pointer reads the business message at buffer position 5, it moves to buffer position 6. The write pointer can be used to move to buffer position 5 and write the business message to buffer position 5.

[0175] See Figure 10 As shown, the write pointer is at buffer position 5 and the read pointer is at buffer position 4. All written business messages have been read through the read pointer, so reading business messages through the read pointer is paused at this point. After the write pointer writes a business message at buffer position 5, it moves to buffer position 6. The read pointer can then be used to move back to buffer position 5 and read the business message at buffer position 5.

[0176] The above examples are only for more clearly illustrating how to pause reading the service message through the reading pointer, continue reading the service message through the reading pointer, pause writing the service message through the writing pointer, and continue writing the service message through the writing pointer, and the present application is not limited thereto.

[0177] In the above technical solution, for any category, the service messages of the category are sequentially written into the buffer positions of the buffer area through the writing pointer according to the buffer position order of the buffer area corresponding to the category, and the service messages are sequentially read through the reading pointer. Thus, the service messages are quickly read, and then the fields in the read service messages that need to be stored in the warehouse are stored in the storage positions corresponding to the category, so as to avoid the influence of deadlock and other problems on the persistence efficiency.

[0178] Figure 11 The third service persistence method provided by the embodiment of the present application has a flowchart of interaction as shown in Figure 11 The flowchart of interaction includes the following steps:

[0179] Step S1101: The online processing device generates a corresponding service message based on the fields that need to be stored in the warehouse in the request information, the information representing the category, and the identification information uniquely identifying the request information.

[0180] Step S1102: The online processing device sends the service message and the message queue information corresponding to the category of the service message to the message queue device.

[0181] Step S1103: The message queue device distributes the service message to the message queue of the corresponding category.

[0182] Step S1104: The warehouse processing device respectively acquires a plurality of service messages of each category from the message queues of a plurality of categories.

[0183] The specific implementation of steps S1101-S1104 can refer to the above embodiment, which will not be repeated here.

[0184] Step S1105: For a plurality of service messages of any category acquired, the warehouse processing device saves the identification information of the plurality of service messages of the category in Redis for a preset time length, and performs deduplication processing on the plurality of service messages of the category based on a data deduplication algorithm of Redis.

[0185] Due to network fluctuations and other factors, the situation of repeated service messages may occur, so the plurality of service messages of each category acquired can be deduplicated.

[0186] For example, the data entry processing device saves the unique identifier of the business message to Redis and uses Redis's data deduplication algorithm to remove duplicates. Due to factors such as network fluctuations, business messages are often duplicated within a short period of time. Therefore, an expiration time for the identifier can be preset. After the identifier is stored in Redis for longer than the preset expiration time, the identifier is deleted to prevent Redis memory from being filled up. This embodiment does not specify a specific expiration time, such as 120s, 100s, or 110s, etc.

[0187] In some optional implementations, in order to ensure the deduplication effect, the inbound processing device sends the identification information that can uniquely identify the business message to the requester corresponding to the business message. The requester notifies the inbound processing device after receiving the same identification information.

[0188] Step S1106: The inbound processing device performs batch persistence processing on multiple business messages of the category based on the persistence method corresponding to the category.

[0189] The specific implementation of step S1106 can be found in the above embodiments, and will not be repeated here.

[0190] In the above technical solution, the duplicate business messages are avoided by deduplicating the business messages.

[0191] See Figure 12 The diagram shown is a third persistent system architecture provided in this embodiment. Figure 1 Based on the persistent system 10 shown, the persistent system 30 also includes a load balancer 500, and the persistent system 30 has multiple online processing devices. Figure 12 Taking online processing devices 101, 102 and 103 as examples, there may be more or fewer online processing devices in actual applications.

[0192] Figure 13 The interaction flowchart of the fourth business persistence method provided in the embodiments of this application is applied to... Figure 12 The persistent system shown, such as Figure 13 As shown, it includes the following steps:

[0193] Step S1301: The load balancer selects one online processing device sequentially from all online processing devices based on the sequential circular queue.

[0194] Step S1302: The load balancer sends the information to be processed to the selected online processing device.

[0195] The to-be-processed information includes the request information, the information representing the category, and the identification information uniquely identifying the request information.

[0196] In an implementation, multiple online processing devices can be provided to improve processing efficiency. An online processing device is selected in sequence based on a sequential circular queue by a load balancer (e.g., an F5 load balancer). The to-be-processed information (including the request information, the information representing the category, and the identification information uniquely identifying the request information) is sequentially and circularly sent to the selected online processing device.

[0197] Since an online processing device cannot generate a service message when it fails, in some optional implementations, if an online processing device fails, the load balancer removes the failed online processing device from the sequential circular queue.

[0198] For example, when an online processing device has a second-layer to seventh-layer failure, the load balancer removes it from the sequential circular queue and does not participate in the next polling, that is, does not send to-be-processed information to the failed online processing device until the online processing device recovers.

[0199] Step S1303: The online processing device generates a corresponding service message based on the field that needs to be stored in the request information, the information representing the category, and the identification information uniquely identifying the request information.

[0200] Step S1304: The online processing device sends the service message and message queue information corresponding to the category of the service message to the message queue device.

[0201] Step S1305: The message queue device distributes the service message to the message queue of the corresponding category.

[0202] Step S1306: The storage processing device respectively acquires multiple service messages of each category from multiple message queues of multiple categories.

[0203] Step S1307: For the multiple service messages of any category, the storage processing device performs batch persistence processing on the multiple service messages of the category based on a persistence mode corresponding to the category.

[0204] The specific implementation of steps S1303-S1307 can refer to the above embodiments, which will not be described here.

[0205] In the embodiments of the present application, the service persistence method performed by the storage processing device is as shown in Figure 14 The service persistence method includes the following steps:

[0206] Step S1401: respectively obtaining a plurality of service messages of each category from a plurality of category message queues; wherein the service messages of the corresponding category are cached in the message queue of each category;

[0207] Step S1402: for a plurality of service messages of any category, based on the corresponding persistence mode of the category, batch processing the plurality of service messages of the category.

[0208] In some optional embodiments, based on the persistence mode corresponding to the category, batch processing the plurality of service messages of the category includes:

[0209] Through the write pointer of the buffer area corresponding to the category, the service messages of the category are sequentially written into the buffer positions of the buffer area in the order of the buffer positions of the buffer area; and through the read pointer of the buffer area, the service messages written through the write pointer are sequentially read in the order of the buffer positions of the buffer area, and the fields that need to be stored in the warehouse in the read service messages are stored in the storage position corresponding to the category.

[0210] In some optional embodiments, further comprising:

[0211] If the write pointer is adjacent to the read pointer and is after the read pointer, then the writing of the service messages of the category into the buffer area through the write pointer is suspended, and after the write pointer is not adjacent to the read pointer, the writing of the service messages of the category into the buffer area through the write pointer is continued; or,

[0212] If the read pointer is adjacent to the write pointer and is after the write pointer, then the reading of the service messages through the read pointer is suspended, and after the read pointer is not adjacent to the write pointer, the reading of the service messages through the read pointer is continued.

[0213] In some optional embodiments, sequentially writing the service messages of the category into the buffer positions of the buffer area includes:

[0214] If the buffer position to be written has service messages, the service messages to be written are overwritten on the existing service messages.

[0215] In some optional embodiments, before batch processing the plurality of service messages of the category based on the persistence mode corresponding to the category, further comprising:

[0216] Saving the identification information of the plurality of service messages of the category in Redis for a preset time length, and performing deduplication processing on the plurality of service messages of the category based on the data deduplication algorithm of Redis.

[0217] In some alternative implementations, for any type of message queue, at least one sub-queue is included;

[0218] The number of sub-queues for any category is adjusted as follows:

[0219] If the ratio between the total number of business messages cached in all sub-queues of the category and the number of sub-queues of the category is greater than a first ratio, then the number of sub-queues of the category is increased; or

[0220] If the ratio between the total number of business messages cached in all sub-queues of the category and the number of sub-queues of the category is less than a second ratio, and the number of sub-queues of the category is greater than a preset number, then the number of sub-queues of the category is reduced.

[0221] Wherein, the first ratio is greater than the second ratio.

[0222] Figure 14 For details on the specific implementation of the embodiments, please refer to the implementation of the above-described interaction method; repeated details will not be repeated here.

[0223] like Figure 15 As shown, based on and Figure 14 Using the same inventive concept as the business persistence method shown, this application provides a first business persistence apparatus 1500, comprising:

[0224] The business message acquisition module 1501 is used to acquire multiple business messages of each category from message queues of multiple categories; wherein, each category of message queue caches business messages of the corresponding category.

[0225] The persistence module 1502 is used to perform batch persistence processing on multiple business messages of any category based on the persistence method corresponding to the category.

[0226] In some optional implementations, the persistence module 1502 is specifically used for:

[0227] Using the write pointer of the buffer corresponding to the category, the business messages of the category are written to the buffer positions of the buffer sequentially according to the buffer position order; and using the read pointer of the buffer, the business messages written by the write pointer are read sequentially according to the buffer position order, and the fields in the read business messages that need to be stored in the database are stored in the storage position corresponding to the category.

[0228] In some alternative implementations, the persistence module 1502 is further configured to:

[0229] If the write pointer is adjacent to the read pointer and after the read pointer, then pausing writing the service messages of the category through the write pointer into the buffer, and continuing writing the service messages of the category through the write pointer into the buffer after the read pointer is not adjacent to the write pointer; or

[0230] If the read pointer is adjacent to the write pointer and after the write pointer, then pausing reading the service messages through the read pointer, and continuing reading the service messages through the read pointer after the read pointer is not adjacent to the write pointer.

[0231] In some optional embodiments, the persistence module 1502 is specifically configured to:

[0232] If the buffered position to be written has a service message, the service message to be written is used to overwrite the existing service message.

[0233] In some optional embodiments, the system further comprises a deduplication module 1503 configured to:

[0234] Before the persistence module 1502 performs batch persistence processing on the plurality of service messages of the category based on the persistence mode corresponding to the category, the identification information of the plurality of service messages of the category is saved in Redis for a preset time length, and the plurality of service messages of the category are subjected to deduplication processing based on a data deduplication algorithm of Redis.

[0235] In some optional embodiments, for any category of message queue, at least one sub-queue is included;

[0236] The number of sub-queues of any category is adjusted in the following manner:

[0237] If the ratio between the total number of service messages cached by all sub-queues of the category and the number of sub-queues of the category is greater than a first ratio, then the number of sub-queues of the category is increased; or

[0238] If the ratio between the total number of service messages cached by all sub-queues of the category and the number of sub-queues of the category is less than a second ratio, and the number of sub-queues of the category is greater than a preset number, then the number of sub-queues of the category is decreased.

[0239] The first ratio is greater than the second ratio.

[0240] Figure 15 The specific implementation of the embodiments can refer to the implementation of the above-mentioned interaction method, and repeated parts will not be described again.

[0241] Based on the same technical concept, the present application also provides a warehousing processing device 1600, as shown inFigure 16 As shown, the device includes at least one processor 1601, and a memory 1602 connected with the at least one processor. The embodiments of the present application do not limit the specific connection medium between the processor 1601 and the memory 1602, Figure 16 For example, the processor 1601 and the memory 1602 are connected through a bus 1603. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 16 In some embodiments, only one thick line is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0242] The processor 1601 is the control center of the warehouse-in processing device, and can be connected with various parts of the warehouse-in processing device through various interfaces and lines, and can realize data processing by running or executing instructions stored in the memory 1602 and calling data stored in the memory 1602. Optionally, the processor 1601 can include one or more processing units, and the processor 1601 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface and the application program, etc., and the modem processor mainly processes the issued instructions. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 1601. In some embodiments, the processor 1601 and the memory 1602 can be implemented on the same chip, and in some embodiments, they can also be respectively implemented on independent chips.

[0243] The processor 1601 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the business persistence method embodiment can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0244] The memory 1602, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 1602 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. The memory 1602 is any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto. The memory 1602 in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, used for storing program instructions and / or data.

[0245] In the embodiments of the present application, the memory 1602 stores a computer program, when the program is executed by the processor 1601, the processor 1601 executes:

[0246] From a plurality of categories of message queues, a plurality of service messages of each category are obtained respectively; wherein the message queue of each category caches the service message of the corresponding category;

[0247] For a plurality of service messages of any category obtained, based on the persistent mode corresponding to the category, a plurality of service messages of the category are batched and persistent processing is performed.

[0248] In some optional embodiments, the processor 1601 specifically executes:

[0249] Through the write pointer of the buffer area corresponding to the category, the service messages of the category are sequentially written into the buffer positions of the buffer area in the order of the buffer positions of the buffer area; and through the read pointer of the buffer area, the service messages written through the write pointer are sequentially read in the order of the buffer positions of the buffer area, and the fields required to be stored in the warehouse in the read service messages are stored in the storage position corresponding to the category.

[0250] In some optional embodiments, the processor 1601 further executes:

[0251] if the write pointer is adjacent to the read pointer and is behind the read pointer, then pausing writing the service messages of the category through the write pointer into the buffer, and continuing writing the service messages of the category through the write pointer into the buffer after the write pointer is not adjacent to the read pointer; or

[0252] if the read pointer is adjacent to the write pointer and is behind the write pointer, then pausing reading the service messages through the read pointer, and continuing reading the service messages through the read pointer after the read pointer is not adjacent to the write pointer.

[0253] In some optional embodiments, the processor 1601 specifically performs:

[0254] if the buffered position to be written has a service message, then the service message to be written is used to overwrite the existing service message.

[0255] In some optional embodiments, the processor 1601 further performs:

[0256] based on the persistence mode corresponding to the category, before performing batch persistence processing on the plurality of service messages of the category, the identification information of the plurality of service messages of the category is saved in Redis for a preset time length, and the plurality of service messages of the category are subjected to deduplication processing based on a data deduplication algorithm of Redis.

[0257] In some optional embodiments, for any category of message queue, at least one sub-queue is included;

[0258] the number of sub-queues of any category is adjusted in the following manner:

[0259] if the ratio between the total number of service messages cached by all sub-queues of the category and the number of sub-queues of the category is greater than a first ratio, then the number of sub-queues of the category is increased; or

[0260] if the ratio between the total number of service messages cached by all sub-queues of the category and the number of sub-queues of the category is less than a second ratio, and the number of sub-queues of the category is greater than a preset number, then the number of sub-queues of the category is decreased;

[0261] wherein the first ratio is greater than the second ratio.

[0262] Since the warehousing processing device is the warehousing processing device in the method in the embodiments of the present application, and the warehousing processing device solves the problem by a similar principle to the method, the implementation of the warehousing processing device can be referred to the implementation of the method, and the repeated parts will not be described herein.

[0263] Based on the same technical concept, the embodiment of the present application further provides a computer readable storage medium, which stores a computer program executable by a computing device, and when the program runs on the computing device, the program causes the computing device to execute the steps of the business persistence method.

[0264] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer-usable program code.

[0265] The present application is described with reference to the flowcharts and / or block diagrams of the method, device (system), and computer program product according to the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0266] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0267] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus, so that a series of operation steps are executed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0268] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that such additions and modifications be included within the scope of the application. It is the following claims, including any amendments thereto, which define the scope of the application.

[0269] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A service persistence method, characterized by, The method is applied to a persistent system, and the persistent system comprises an online processing device, a message queue device and an ingesting processing device; the method comprises the following steps: The online processing device generates a corresponding business message based on the fields needing to be ingested in the request information, the information representing the category and the identification information uniquely identifying the request information; the fields needing to be ingested corresponding to different categories of request information are different; The online processing device sends the business message and the message queue information corresponding to the category of the business message to the message queue device; The message queue device distributes the business message to the message queue of the corresponding category based on the message queue information corresponding to the category of the business message; The ingesting processing device respectively acquires a plurality of business messages of each category from a plurality of categories of message queues; wherein the message queue of each category caches the business messages of the corresponding category; The ingesting processing device performs batch persistent processing on the plurality of business messages of the category based on the persistent mode corresponding to the category, including: sequentially writing the business messages of the category into the buffer positions of the buffer area in the order of the buffer positions of the buffer area through the write pointer of the buffer area; and sequentially reading the business messages written through the write pointer in the order of the buffer positions of the buffer area through the read pointer of the buffer area, and storing the fields needing to be ingested in the read business messages into the storage position corresponding to the category.

2. The method of claim 1, wherein, Further comprising: If the write pointer is adjacent to the read pointer and is behind the read pointer, the ingesting processing device suspends writing the business messages of the category into the buffer area through the write pointer, and continues to write the business messages of the category into the buffer area through the write pointer after the write pointer is not adjacent to the read pointer; Or, If the read pointer is adjacent to the write pointer and is behind the write pointer, the ingesting processing device suspends reading the business messages through the read pointer, and continues to read the business messages through the read pointer after the read pointer is not adjacent to the write pointer.

3. The method of claim 1, wherein, Sequentially writing the business messages of the category into the buffer positions of the buffer area comprises: If the buffer position to be written has a business message, the business message to be written is overlaid on the existing business message.

4. The method of claim 1, wherein, Before performing batch persistent processing on the plurality of business messages of the category based on the persistent mode corresponding to the category, the method further comprises the following steps: The ingesting processing device saves the identification information of the plurality of business messages of the category in a remote dictionary service Redis for a preset time length, and performs deduplication processing on the plurality of business messages of the category based on a data deduplication algorithm of Redis.

5. The method according to any one of claims 1 to 4, characterized in that, For any category of message queue, at least one sub-queue is included; The number of sub-queues of any category is adjusted in the following manner: If the ratio between the total number of business messages cached in all sub-queues of the category and the number of sub-queues of the category is greater than a first ratio, the number of sub-queues of the category is increased; or if a ratio between a total number of service messages cached by all sub-queues of the category and a number of sub-queues of the category is less than a second ratio, and the number of sub-queues of the category is greater than a preset number, then reducing the sub-queues of the category; wherein the first ratio is greater than the second ratio.

6. The method of any one of claims 1 to 4, wherein, The persistent system further comprises a load balancer, and there are multiple online processing devices in the persistent system; before the online processing device generates a corresponding service message based on the field needing to be warehoused in the request information, the information representing the category, and the identification information uniquely identifying the request information, the method comprises: The load balancer sequentially selects an online processing device from all online processing devices based on the sequential circular queue, and sends the to-be-processed information to the selected online processing device; wherein the to-be-processed information comprises the request information, the information representing the category, and the identification information uniquely identifying the request information.

7. The method of claim 6, wherein, The method further comprises: If an online processing device fails, the load balancer removes the failed online processing device from the sequential circular queue.

8. A service persistence method, characterized by, Applied to a warehousing processing device, the method comprises: From multiple category message queues, multiple service messages of each category are obtained respectively; wherein each category message queue caches service messages of the corresponding category; For multiple service messages of any category obtained, based on the persistent mode corresponding to the category, batch persistent processing is performed on the multiple service messages of the category, including: through the write pointer of the buffer area corresponding to the category, the service messages of the category are sequentially written into the buffer positions of the buffer area in the order of the buffer positions of the buffer area; and through the read pointer of the buffer area, the service messages written through the write pointer are sequentially read in the order of the buffer positions of the buffer area, and the fields needing to be warehoused in the read service messages are stored in the storage position corresponding to the category.

9. A service persistence apparatus characterized by comprising: Comprise: A service message obtaining module is configured to obtain multiple service messages of each category from multiple category message queues respectively; wherein each category message queue caches service messages of the corresponding category; A persistent module is configured to perform batch persistent processing on multiple service messages of any category based on the persistent mode corresponding to the category; the persistent module is specifically configured to: through the write pointer of the buffer area corresponding to the category, the service messages of the category are sequentially written into the buffer positions of the buffer area in the order of the buffer positions of the buffer area; and through the read pointer of the buffer area, the service messages written through the write pointer are sequentially read in the order of the buffer positions of the buffer area, and the fields needing to be warehoused in the read service messages are stored in the storage position corresponding to the category.

Images