Data processing method and equipment based on multi-message queue system

Through a heterogeneous multi-message queue system combined with message queue interface and hierarchical storage management in different scenarios, the problem of the performance and cost of message queue system under high service data volume in the existing technology is solved, and efficient and low-cost data processing and storage.

CN119025298APending Publication Date: 2024-11-26BEIJING VOLCANO ENGINE TECH CO LTD
View PDF 12 Cites 0 Cited by

Patent Information

Application Number
CN202411148250.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

When existing message queue systems process high service data volume, it is difficult to take into account high performance and low cost, and data access efficiency and resource utilization efficiency in different scenarios are insufficient.

Method used

Adopt heterogeneous multi-message queue system, using different message queue system interfaces according to instant service and intensive service scenarios, combining the advantages of low-latency and high-throughput message queue systems, providing a hybrid storage solution, and optimizing data access through hierarchical storage management services. Strategy.

Benefits of technology

It improves the processing efficiency of data requests, reduces the cost and overhead of storage resources, provides a higher performance and low-cost message queue system, and enhances data reliability and storage management capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025298A_ABST
    Figure CN119025298A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a data processing method and device based on a multi-message queue system, and the method comprises the steps: receiving a data reading request which is used for requesting to read first target data; if the data reading request belongs to the instant service scene, reading first target data from a first message queue system for response by using a data access interface of the first message queue system in the heterogeneous multi-message queue system; and if the data reading request belongs to the dense service scene, using a data access interface of a second message queue system in the heterogeneous multi-message queue system to read the first target data from the second message queue system for response. According to the method, different data access interfaces can be provided for the outside in combination with different business demand scenes, and the processing efficiency of the data request is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The disclosed embodiments relate to the field of cloud network technology, and in particular to a data processing method and device based on a multi-message queue system. Background Art

[0002] A message is a unit of data sent between two computers. A message can be very simple, such as containing only a text string, or it can be more complex and may contain embedded objects. The message queue system is a container that stores messages during their transmission. Messages can be sent to a message queue system, which needs to provide storage and delivery functions for the messages. The main purpose of a message queue system is to provide message routing and guarantee message delivery. If the recipient is unavailable when a message is sent, the message queue will retain the message until it can be successfully delivered.

[0003] As the amount of business data continues to grow, business applications have also placed higher demands on the processing efficiency and resource overhead of the message queue system. Summary of the invention

[0004] The embodiments of the present disclosure provide a data processing method and device based on a multi-message queue system, which can improve the processing efficiency of data requests.

[0005] In a first aspect, an embodiment of the present disclosure provides a data processing method based on a multi-message queue system, comprising:

[0006] receiving a data read request, wherein the data read request is used to request to read first target data;

[0007] If the data reading request belongs to an instant service scenario, using a data access interface of a first message queue system in a heterogeneous multi-message queue system, read the first target data from the first message queue system for response;

[0008] If the data read request belongs to an intensive service scenario, the data access interface of the second message queue system in the heterogeneous multi-message queue system is used to read the first target data from the second message queue system for response.

[0009] In a second aspect, an embodiment of the present disclosure provides a data processing device based on a multi-message queue system, including:

[0010] A receiving unit, configured to receive a data read request, wherein the data read request is configured to request to read first target data;

[0011] A processing unit, configured to, if the data reading request belongs to an instant service scenario, use a data access interface of a first message queue system in a heterogeneous multi-message queue system to read first target data from the first message queue system for response;

[0012] The processing unit is further used to, if the data reading request belongs to an intensive service scenario, use the data access interface of the second message queue system in the heterogeneous multi-message queue system to read the first target data from the second message queue system in response.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0014] The memory stores computer-executable instructions;

[0015] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the data processing method based on the multi-message queue system as described in the first aspect and various possible designs of the first aspect.

[0016] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the data processing method based on a multi-message queue system as described in the first aspect and various possible designs of the first aspect is implemented.

[0017] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the data processing method based on a multi-message queue system as described in the first aspect and various possible designs of the first aspect.

[0018] The data processing method and device based on the multi-message queue system provided in this embodiment include: receiving a data read request, the data read request is used to request to read the first target data; if the data read request belongs to an instant service scenario, the data access interface of the first message queue system in the heterogeneous multi-message queue system is used to read the first target data from the first message queue system for response; if the data read request belongs to an intensive service scenario, the data access interface of the second message queue system in the heterogeneous multi-message queue system is used to read the first target data from the second message queue system for response. In this technical solution, since the data access interface of the first message queue system can be used for the instant service scenario, the first target data can be read from the first message queue system for response, and the data access interface of the second message queue system can be used for the intensive service scenario, the first target data can be read from the second message queue system for response, which realizes the combination of different business demand scenarios, and provides different data access interfaces to the outside, and the processing efficiency of data requests can be improved through multiple data access interfaces; and, since the design advantages of different message queue systems are combined, the advantages of each of the multiple message queue systems are utilized to effectively open up the multiple message queue systems, providing a more high-performance, low-cost hybrid storage message queue system, thereby saving storage resources and reducing cost overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1 A schematic diagram of an application scenario of a data processing method based on a multi-message queue system provided in an embodiment of the present disclosure;

[0021] Figure 2 The process of the data processing method based on the multi-message queue system provided by the embodiment of the present disclosure Figure 1 ;

[0022] Figure 3 A schematic diagram of a data processing method based on a multi-message queue system provided in an embodiment of the present disclosure Figure 1 ;

[0023] Figure 4 A schematic diagram of a data processing method based on a multi-message queue system provided in an embodiment of the present disclosure Figure 2 ;

[0024] Figure 5A schematic diagram of a data processing method based on a multi-message queue system provided in an embodiment of the present disclosure Figure 3 ;

[0025] Figure 6 A schematic diagram of the structure of a data processing device based on a multi-message queue system provided in an embodiment of the present disclosure;

[0026] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0028] A message (a type of business data in this application) is a unit of data transmitted between two computers. A message can be very simple, such as containing only a text string; or it can be more complex and may contain embedded objects. The message queue system is a container that stores messages during the transmission of the message. Messages can be sent to a message queue system, which needs to provide storage and delivery functions for the messages. The main purpose of a message queue system is to provide message routing and ensure message delivery. If the recipient is unavailable when the message is sent, the message queue will retain the message until it can be successfully delivered.

[0029] The current message queue system generally includes a low-latency message queue system and a high-throughput message queue system. Among them, the low-latency message queue system can ensure lower write latency and generally uses solid-state drives for storage. Among them, the high-throughput message queue system generally uses mechanical hard drives for storage and has the performance advantage of high throughput.

[0030] However, at the beginning of the design of different message systems, the architecture is usually designed according to its design goals, which makes each message queue system have its unique advantages and disadvantages.

[0031] For example, the Apache Kafka messaging system in the low-latency messaging queue system was originally designed to solve the problem of reliable transmission of massive data in big data scenarios, so that it can also have high throughput performance advantages when using relatively cheap mechanical hard disk storage hardware. However, its architectural design results in a relatively limited number of Topics carried by the Apache Kafka messaging system. When the number of Topics is too large, the cluster performance will drop sharply.

[0032] For example, the Apache RocketMQ messaging system in the high-throughput message queue system is designed for low-latency online application scenarios such as e-commerce and microservices. While ensuring throughput as much as possible, it has a unique architecture design specifically for multi-topic scenarios, so that cluster performance will not drop rapidly with the number of topics. In order to ensure lower write latency, the Apache RocketMQ messaging system mostly uses more expensive solid-state drives for storage, which is costly.

[0033] As the amount of business data continues to grow, business applications have also placed higher demands on the processing efficiency and resource overhead of the message queue system.

[0034] In view of the technical problems in the prior art, the inventors have come up with the following technical ideas: combining the data layered storage methods of various message queues, combining the design advantages of different message queue systems, and providing a more high-performance, low-cost hybrid storage message queue system. While reducing the data storage cost, it also makes more reasonable use of the architectural advantages of various different message queue systems, and expands the system functions in the original message queue system.

[0035] Correspondingly, the specific steps include: receiving a data read request, the data read request is used to request to read the first target data; if the data read request belongs to an instant service scenario, using the data access interface of the first message queue system in the heterogeneous multi-message queue system, reading the first target data from the first message queue system in response; if the data read request belongs to an intensive service scenario, using the data access interface of the second message queue system in the heterogeneous multi-message queue system, reading the first target data from the second message queue system in response.

[0036] In this technical solution, since the data access interface of the first message queue system can be used for instant service scenarios, the first target data can be read from the first message queue system for response, and for intensive service scenarios, the data access interface of the second message queue system can be used to read the first target data from the second message queue system for response, and the processing efficiency of data requests can be improved through multiple data access interfaces; and since the design advantages of different message queue systems are combined and the respective advantages of multiple message queue systems are utilized, the multiple message queue systems are effectively connected, providing a higher performance, low-cost hybrid storage message queue system, thereby saving storage resources and reducing cost overhead.

[0037] The application scenarios of the embodiments of the present disclosure are explained below:

[0038] In the multi-message queue system of the embodiment of the present disclosure, as Figure 1As shown, the multi-message queue system includes: an application layer, an access layer and a storage layer. The access layer and the storage layer are the main contents of the present invention, and the application layer is a demonstration of the usage.

[0039] In the application layer, the data access mode (pattern) can be flexibly selected according to the business needs of the business party. In the disclosed embodiment, the multi-message queue system can select a low-latency data access mode or a high-throughput data access mode according to the business needs scenario of the business party.

[0040] In the access layer, there are three data interaction modes: low-latency data write mode, low-latency data access mode, and high-throughput data access mode. Among them, the low-latency data write mode is used for low-latency writing of data. Among them, the low-latency data access mode is used for low-latency reading of data. Among them, the high-throughput data access mode is used to read multiple data at a time.

[0041] In the storage layer, it is divided into three parts: the first message queue system, the second message queue system, and the hierarchical storage management service. Among them, the first message queue system is deployed with high-performance and high-cost solid-state hard disks to provide low-latency and high-performance data storage services. The second message queue system is deployed with low-cost mechanical hard disks to provide larger storage space and low-cost data storage services. The hierarchical storage management service manages data between the two storage systems. Its main functions are: periodically writing the data of the first message queue system to the second message queue system; managing the correlation between the data of the first message queue system and the second message queue system; and providing the first message queue system with the function of accessing historical data through the first message queue system.

[0042] One point that needs to be further explained is that in this multi-message queue system, by combining the two message queue systems, more heterogeneous storage copies are provided for business data, which increases the reliability of business data and reduces the problem of data loss caused by hardware failure of a single system.

[0043] By combining the two message queue systems, the following problems can be solved:

[0044] (1) Achieve a good balance between high performance and low cost. High performance and low cost can be taken into account in the same system, and different usage solutions can be provided according to different usage scenarios.

[0045] (2) Low message storage cost per unit time: data is divided into hot data that requires high-performance access and cold data that requires low-cost storage, thus reducing the overall data storage cost. A low-cost long-term data storage solution is provided.

[0046] (3) Using the same system, two data consumption scenarios, low-latency data consumption scenario and high-throughput data consumption scenario, are provided. For instant service scenarios (e.g., low-latency data consumption scenarios), the data access interface of the first message queue system can be used to provide fast data access. For intensive service scenarios (e.g., high-throughput data consumption scenarios), the data access interface of the second message queue system can be used to provide high-throughput data access.

[0047] (4) Based on the original message system, a heterogeneous message storage system is used to provide higher data reliability.

[0048] The following is a detailed description of the data processing method based on the multi-message queue system through a specific embodiment:

[0049] Figure 2 The data processing method based on the multi-message queue system provided by the embodiment of the present disclosure is as follows: Figure 1 In the embodiment of the present disclosure, the execution subject of the data request processing method may be an electronic device, and the electronic device may be a terminal or a server. Figure 2 As shown, the data processing method based on the multi-message queue system includes:

[0050] S201: Receive a data read request, where the data read request is used to request to read first target data.

[0051] In the embodiment of the present disclosure, the data read request may be a data read request sent by a first service, and is used to read first target data from a multi-message queue system.

[0052] The first service may be a service executed in an application. For example, the first service may be an online order service, and the first target data corresponding thereto may be order information. For another example, the first service may be an online alarm service, and the first target data corresponding thereto may be the transaction duration of a submitted (unsuccessful) order, etc.

[0053] In some embodiments, the service scenario to which the data read request belongs can be determined based on the service identifier. Optionally, the data read request carries a target service identifier. Accordingly, after receiving the data read request, it also includes: obtaining the target service identifier from the data read request, and determining that the service scenario to which the data read request belongs is an instant service scenario or an intensive service scenario based on the target service identifier.

[0054] Optionally, the instant service scenario includes a low-latency data access mode, and the intensive service scenario includes a high-throughput data access mode.

[0055] Optionally, the service scenario to which the data read request belongs is determined to be an instant service scenario or an intensive service scenario based on the target business identifier, including: if the target business identifier indicates an online business, then determining that the data read request belongs to the instant service scenario; if the target business identifier indicates an offline business, then determining that the data read request belongs to the intensive service scenario.

[0056] For example, Figure 1 As shown, for example, the target business identifier is an online application, which belongs to the online business (that is, it indicates an online business), and it is determined that the data read request belongs to the instant service scenario. The target business identifier is the online alarm business of the alarm application, which indicates the online business, and it is determined that the data read request belongs to the instant service scenario. For another example, the target business identifier is the offline data warehouse business, which indicates the offline business, and it is determined that the data read request belongs to the intensive service scenario. The target business identifier is the alarm clustering business of the alarm application, which indicates the offline business, and it is determined that the data read request belongs to the intensive service scenario.

[0057] In the disclosed embodiments, different data access modes and data access strategies can be provided externally in combination with different business service scenarios. For low-latency online applications, a high-performance data access mode can be provided; for offline big data scenarios, a high-throughput data access mode can be provided, which realizes the combination of the design advantages of different message queue systems, and effectively connects multiple message queue systems by utilizing their respective advantages, providing a more high-performance, low-cost hybrid storage message queue system, saving storage resources and reducing cost overhead.

[0058] S202: If the data reading request belongs to an instant service scenario, use the data access interface of the first message queue system in the heterogeneous multi-message queue system to read the first target data from the first message queue system for response.

[0059] In the disclosed embodiment, the heterogeneous multi-message queue system includes: an application layer, an access layer and a storage layer. The storage layer includes: a first message queue system, a second message queue system and a hierarchical storage management service. Optionally, data management is performed between the first message queue system and the second message queue system through the hierarchical storage management service.

[0060] It should be noted that the business data stored in the first message queue system is called local data, and the data stored in the second message queue system is called remote data. The first message queue system is called the local storage system, and the second message queue system is called the remote storage system.

[0061] Among them, the instant service scenario includes a low-latency data access mode. In the low-latency data access mode, it can be divided into: data access strategy based on local and remote data and data access strategy based on hot and cold data according to different access strategies.

[0062] The following describes these two data access strategies respectively through specific embodiments:

[0063] In some embodiments, based on local and remote data access strategies, the data access interface of the first message queue system in the heterogeneous multi-message queue system may be used to read the first target data from the first message queue system for response.

[0064] Alternatively, if Figure 3 As shown, using the data access interface of the first message queue system in the heterogeneous multi-message queue system, the first target data is read from the first message queue system for response, including: using the data access interface of the first message queue system in the heterogeneous multi-message queue system, adopting a local and remote-based data access strategy, preferentially reading the first target data from the local storage of the first message queue system, and when the data is insufficient, reading the data from the local storage of the second message queue system and responding.

[0065] It should be noted that in order to further reduce the delay in reading data from the local storage of the message queue system (including the first message queue system and the second message queue system mentioned above), the data access interface of the first message queue system can be used in the present application to read the first target data by reading a single data.

[0066] In this embodiment, in the data access strategy based on local and remote, if the accessed data is stored in the local storage system (i.e., the first message queue system), it is obtained from the local storage system. When the data is not stored in the local storage system, it is attempted to read the data in the remote storage system (i.e., the second message queue system). In this mode, the data in the first message queue system will be accessed first, and the hardware performance of the first message queue system is relatively high, which can ensure the maximization of data access performance.

[0067] In other embodiments, based on the data access strategy of hot and cold data, the data access interface of the first message queue system in the heterogeneous multi-message queue system can be used to read the first target data from the first message queue system for response. Among them, the data that can be hit by accessing the cache is called "hot data", and the data that cannot be hit by accessing the cache is called "cold data".

[0068] Alternatively, if Figure 4As shown, using the data access interface of the first message queue system in the heterogeneous multi-message queue system, reading the first target data from the first message queue system for response, including: using the data access interface of the first message queue system in the heterogeneous multi-message queue system, adopting a data access strategy based on cold and hot data, preferentially reading the first target data from the cache of the first message queue system, and when the data is insufficient, reading the data from the local storage of the first message queue system for response; wherein the cache of the first message queue system is used to store hot data.

[0069] In this embodiment, the cache of the local storage system is accessed first, and if the cache is hit, the cache data is read first. If it is not hit, the local storage system is read, which can reduce the disk data of reading local data, thereby reducing the load of the first message queue system.

[0070] It should be noted that when the data in the cache of the first message queue system is insufficient, data can also be directly read from the local storage of the second message queue system for response.

[0071] In this case, in the data access strategy based on hot and cold data, the cache of the local storage system will be accessed first. If the cache is hit, the cache data will be read first. If it cannot be hit, the remote storage system data will be directly tried to be read, further reducing the disk data for reading local data, thereby further reducing the load of the first message queue system.

[0072] In the embodiments of the present disclosure, the local storage system and the remote storage system can provide load isolation for hot and cold data, thereby avoiding the situation where the overall performance of the cluster is affected by abnormal access to some data, thereby improving the stability of the multi-message queue system.

[0073] It should be noted that, in the disclosed embodiment, the association between the data of the first message queue system and the second message queue system can be managed through the hierarchical storage management service. That is, the correspondence between the business data and the data identifier in the first message queue system can be made consistent with the correspondence between the business data and the data identifier in the second message queue system through the hierarchical storage management service. For example, if the data identifier corresponding to the business data A in the first message queue system is 001, then the data identifier corresponding to the business data A in the second message queue system is 001. In this way, it is possible to access historical business data from the second message queue system through the first message queue system.

[0074] Optionally, in the above two access strategies, data can be read from the local storage of the second message queue system by reading a single piece of data through the hierarchical storage management service and a response can be made, specifically including: determining the data identifier of the first target data through the hierarchical storage management service; and reading the first target data from the local storage of the second message queue system by reading a single piece of data according to the data identifier of the first target data and a response can be made.

[0075] S203: If the data read request belongs to an intensive service scenario, use the data access interface of the second message queue system in the heterogeneous multi-message queue system to read the first target data from the second message queue system for response.

[0076] In the embodiment of the present disclosure, the first message queue system and the second message queue system are deployed using different hardware. The hard disk performance used by the first message queue system is higher than the hard disk performance used by the second message queue system. Figure 1 As shown, the first message queue system is deployed using a solid-state hard disk, and the second message queue system is deployed using a mechanical hard disk.

[0077] It should be noted that the first message queue system and the second message queue system may be various types of message queue systems, and by combining the first message queue system and the second message queue system, a stronger system function may be achieved.

[0078] In some embodiments, the first message queue system is deployed with a high-performance and high-cost solid-state hard disk to provide low-latency and high-performance data storage services. The second message queue system is deployed with a low-cost mechanical hard disk to provide a larger storage space and low-cost data storage service. Optionally, the first message queue system can be a RocketMQ (Rocket Message Queue) storage system, and the second message queue system can be a Kafka storage system.

[0079] In the embodiments of the present disclosure, Figure 5 As shown, the intensive service scenario includes a high-throughput data access mode. If the data read request belongs to the high-throughput data access mode, the data access interface of the second message queue system in the heterogeneous multi-message queue system is used to read the first target data from the second message queue system for response.

[0080] It should be noted that in the high-throughput data access mode, the application layer pays more attention to the data access throughput, that is, it can read more consumption data per second. At this time, the business data can be directly accessed through the data access interface of the second message queue system, which can reduce the data storage cost and realize the high-throughput feature that the first message queue system does not have.

[0081] The disclosed embodiment provides a data processing method based on a multi-message queue system: receiving a data read request, the data read request is used to request to read the first target data; if the data read request belongs to an instant service scenario, the data access interface of the first message queue system in the heterogeneous multi-message queue system is used to read the first target data from the first message queue system in response; if the data read request belongs to an intensive service scenario, the data access interface of the second message queue system in the heterogeneous multi-message queue system is used to read the first target data from the second message queue system in response. In this technical solution, since the data access interface of the first message queue system can be used to read the first target data from the first message queue system in response for the instant service scenario, and the data access interface of the second message queue system can be used to read the first target data from the second message queue system in response for the intensive service scenario, it is possible to realize the combination of different business demand scenarios and provide different data access interfaces to the outside, thereby improving the processing efficiency of data requests.

[0082] It should be noted that when a data write request for any business data is received, the data can be written through the data access interface of the first message queue system. Accordingly, the method also includes: receiving a data write request through the data access interface of the first message queue system in the heterogeneous multi-message queue system, the data write request is used to request to write the second target data; writing the second target data to the local storage of the first message queue system; calling the hierarchical storage management service to write the message queue based on the writing method of the specified offset, and writing the second target data to the local storage of the second message queue system.

[0083] In the above situation, since the first message queue system uses the hardware deployment of the solid state drive, which has better write throughput and write latency, the second target data is written to the local storage of the first message queue system, which has higher write performance and lower write latency.

[0084] In the embodiment of the present disclosure, the business data in the first message queue system can also be managed through the hierarchical storage management service. For example, the hardware and architecture differences of different message queue systems can be used to implement a more inexpensive and stable storage service.

[0085] In some embodiments, the hierarchical storage management service can also manage the business data according to the storage duration of the business data. The corresponding specific steps may include: calling the hierarchical storage management service to control the second message queue system to maintain the data whose storage duration is longer than the preset time threshold, and deleting the data whose storage duration is longer than the preset time threshold from the first message queue system; the second message queue system supports the first message queue system to access the full amount of data.

[0086] The preset time threshold may be set according to the specific situation. For example, the preset time threshold may be 2 days, 3 days, 7 days, etc.

[0087] In the disclosed embodiment, data with a longer period can be saved in the second message queue system through the hierarchical storage management service, wherein the second message queue system can use low-cost mechanical hard disks to store data with a longer period, thereby achieving a low-cost and longer storage period architecture design.

[0088] It should be noted that after calling the hierarchical storage management service to synchronize data with a storage time longer than a preset time threshold in the first message queue system to the second message queue system, the data can be deleted synchronously in the first message queue system, or it can be deleted asynchronously and only deleted after the storage time exceeds another preset time threshold.

[0089] Exemplarily, the preset time threshold may be 2 days, and another preset time threshold is set to 7 days. After synchronizing data with a storage time longer than 2 days to the second message queue system, the data may be deleted asynchronously. The data is deleted only after the storage time of the data exceeds 7 days.

[0090] In this case, the system can provide more heterogeneous storage copies by combining the two storage systems, increase data reliability, and reduce data loss caused by hardware failure of a single system.

[0091] Figure 6 A structural diagram of a data processing device based on a multi-message queue system provided in an embodiment of the present disclosure, such as Figure 6 As shown, the data request processing device includes:

[0092] The receiving unit 601 is used to receive a data read request, where the data read request is used to request to read first target data;

[0093] Processing unit 602 is used to read first target data from the first message queue system in the heterogeneous multi-message queue system to respond if the data reading request belongs to an instant service scenario;

[0094] The processing unit 602 is further configured to, if the data read request belongs to an intensive service scenario, use a data access interface of a second message queue system in the heterogeneous multi-message queue system to read the first target data from the second message queue system in response.

[0095] According to one or more embodiments of the present disclosure, if the data reading request carries a target service identifier, the device further includes: a determination unit; the determination unit is used to obtain the target service identifier from the data reading request, and determine whether the service scenario to which the data reading request belongs is an instant service scenario or an intensive service scenario according to the target service identifier.

[0096] According to one or more embodiments of the present disclosure, the determination unit is used to determine that the data reading request belongs to an instant service scenario if the target service identifier indicates an online service; if the target service identifier indicates an offline service, determine that the data reading request belongs to an intensive service scenario.

[0097] According to one or more embodiments of the present disclosure, the device also includes: a data receiving unit; the data receiving unit is used to receive a data write request through a data access interface of a first message queue system in the heterogeneous multi-message queue system, wherein the data write request is used to request to write second target data; write the second target data to the local storage of the first message queue system; call a hierarchical storage management service to write the message queue based on a writing method of a specified offset, and write the second target data to the local storage of the second message queue system.

[0098] According to one or more embodiments of the present disclosure, the device also includes: a data scheduling unit; the data receiving unit is used to call the hierarchical storage management service to control the second message queue system to maintain data with a storage duration greater than a preset time threshold, and delete the data with a storage duration greater than the preset time threshold from the first message queue system; the second message queue system supports the first message queue system to access the full amount of data.

[0099] According to one or more embodiments of the present disclosure, the processing unit 602 uses the data access interface of the first message queue system in the heterogeneous multi-message queue system to read the first target data from the first message queue system in response, specifically including: using the data access interface of the first message queue system in the heterogeneous multi-message queue system, adopting a local and remote-based data access strategy, preferentially reading the first target data from the local storage of the first message queue system, and when the data is insufficient, reading the data from the local storage of the second message queue system and responding.

[0100] According to one or more embodiments of the present disclosure, the processing unit 602 uses the data access interface of the first message queue system in the heterogeneous multi-message queue system to read the first target data from the first message queue system in response, specifically including: using the data access interface of the first message queue system in the heterogeneous multi-message queue system, adopting a data access strategy based on cold and hot data, preferentially reading the first target data from the cache of the first message queue system, and when the data is insufficient, reading the data from the local storage of the first message queue system in response; wherein the cache of the first message queue system is used to store hot data.

[0101] According to one or more embodiments of the present disclosure, the first message queue system and the second message queue system adopt different hardware deployments; wherein the hard disk performance adopted by the first message queue system is higher than the hard disk performance adopted by the second message queue system.

[0102] According to one or more embodiments of the present disclosure, the first message queue system is deployed using a solid-state hard disk, and the second message queue system is deployed using a mechanical hard disk.

[0103] refer to Figure 7 , which shows a schematic diagram of the structure of an electronic device 700 suitable for implementing the embodiment of the present disclosure, and the electronic device 700 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0104] like Figure 7As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. Various programs and data required for the operation of the electronic device 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0105] Typically, the following devices may be connected to the I / O interface 705: input devices 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 708 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 709. The communication devices 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0106] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0107] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0108] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0109] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0110] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0111] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0112] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0113] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0114] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0115] In a first aspect, according to one or more embodiments of the present disclosure, a data processing method based on a multi-message queue system is provided, comprising:

[0116] receiving a data read request, wherein the data read request is used to request to read first target data;

[0117] If the data reading request belongs to an instant service scenario, using a data access interface of a first message queue system in a heterogeneous multi-message queue system, read the first target data from the first message queue system for response;

[0118] If the data read request belongs to an intensive service scenario, the data access interface of the second message queue system in the heterogeneous multi-message queue system is used to read the first target data from the second message queue system for response.

[0119] According to one or more embodiments of the present disclosure, the data reading request carries a target service identifier, and the method further includes: obtaining the target service identifier from the data reading request, and determining, based on the target service identifier, whether the service scenario to which the data reading request belongs is an instant service scenario or an intensive service scenario.

[0120] According to one or more embodiments of the present disclosure, if the target service identifier indicates an online service, it is determined that the data read request belongs to an instant service scenario; if the target service identifier indicates an offline service, it is determined that the data read request belongs to an intensive service scenario.

[0121] According to one or more embodiments of the present disclosure, the method also includes: receiving a data write request through a data access interface of a first message queue system in the heterogeneous multi-message queue system, wherein the data write request is used to request writing of second target data; writing the second target data to the local storage of the first message queue system; and calling a hierarchical storage management service to write the message queue based on a write method of a specified offset, and writing the second target data to the local storage of the second message queue system.

[0122] According to one or more embodiments of the present disclosure, the method also includes: calling a hierarchical storage management service to control the second message queue system to maintain data whose storage duration is greater than a preset time threshold, and deleting the data whose storage duration is greater than the preset time threshold from the first message queue system; the second message queue system supports the first message queue system to access the full amount of data.

[0123] According to one or more embodiments of the present disclosure, the use of the data access interface of the first message queue system in the heterogeneous multi-message queue system to read the first target data from the first message queue system in response includes: using the data access interface of the first message queue system in the heterogeneous multi-message queue system, adopting a local and remote-based data access strategy, preferentially reading the first target data from the local storage of the first message queue system, and when the data is insufficient, reading the data from the local storage of the second message queue system and responding.

[0124] According to one or more embodiments of the present disclosure, the use of the data access interface of the first message queue system in the heterogeneous multi-message queue system to read the first target data from the first message queue system in response includes: using the data access interface of the first message queue system in the heterogeneous multi-message queue system, adopting a data access strategy based on cold and hot data, preferentially reading the first target data from the cache of the first message queue system, and when the data is insufficient, reading the data from the local storage of the first message queue system in response; wherein the cache of the first message queue system is used to store hot data.

[0125] According to one or more embodiments of the present disclosure, the first message queue system and the second message queue system adopt different hardware deployments; wherein the hard disk performance adopted by the first message queue system is higher than the hard disk performance adopted by the second message queue system.

[0126] According to one or more embodiments of the present disclosure, the first message queue system is deployed using a solid-state hard disk, and the second message queue system is deployed using a mechanical hard disk.

[0127] In a second aspect, according to one or more embodiments of the present disclosure, a data processing device based on a multi-message queue system is provided, comprising:

[0128] A receiving unit, configured to receive a data read request, wherein the data read request is configured to request to read first target data;

[0129] A processing unit, configured to, if the data reading request belongs to an instant service scenario, use a data access interface of a first message queue system in a heterogeneous multi-message queue system to read first target data from the first message queue system for response;

[0130] The processing unit is further used to, if the data reading request belongs to an intensive service scenario, use the data access interface of the second message queue system in the heterogeneous multi-message queue system to read the first target data from the second message queue system in response.

[0131] According to one or more embodiments of the present disclosure, if the data reading request carries a target service identifier, the device further includes: a determination unit; the determination unit is used to obtain the target service identifier from the data reading request, and determine whether the service scenario to which the data reading request belongs is an instant service scenario or an intensive service scenario according to the target service identifier.

[0132] According to one or more embodiments of the present disclosure, the determination unit is used to determine that the data reading request belongs to an instant service scenario if the target service identifier indicates an online service; if the target service identifier indicates an offline service, determine that the data reading request belongs to an intensive service scenario.

[0133] According to one or more embodiments of the present disclosure, the device also includes: a data receiving unit; the data receiving unit is used to receive a data write request through a data access interface of a first message queue system in the heterogeneous multi-message queue system, wherein the data write request is used to request to write second target data; write the second target data to the local storage of the first message queue system; call a hierarchical storage management service to write the message queue based on a writing method of a specified offset, and write the second target data to the local storage of the second message queue system.

[0134] According to one or more embodiments of the present disclosure, the device also includes: a data scheduling unit; the data receiving unit is used to call the hierarchical storage management service to control the second message queue system to maintain data with a storage duration greater than a preset time threshold, and delete the data with a storage duration greater than the preset time threshold from the first message queue system; the second message queue system supports the first message queue system to access the full amount of data.

[0135] According to one or more embodiments of the present disclosure, the processing unit uses the data access interface of the first message queue system in the heterogeneous multi-message queue system to read the first target data from the first message queue system in response, which specifically includes: using the data access interface of the first message queue system in the heterogeneous multi-message queue system, adopting a local and remote-based data access strategy, preferentially reading the first target data from the local storage of the first message queue system, and when the data is insufficient, reading the data from the local storage of the second message queue system and responding.

[0136] According to one or more embodiments of the present disclosure, the processing unit uses the data access interface of the first message queue system in the heterogeneous multi-message queue system to read the first target data from the first message queue system in response, specifically including: using the data access interface of the first message queue system in the heterogeneous multi-message queue system, adopting a data access strategy based on cold and hot data, preferentially reading the first target data from the cache of the first message queue system, and when the data is insufficient, reading the data from the local storage of the first message queue system in response; wherein, the cache of the first message queue system is used to store hot data.

[0137] According to one or more embodiments of the present disclosure, the first message queue system and the second message queue system adopt different hardware deployments; wherein the hard disk performance adopted by the first message queue system is higher than the hard disk performance adopted by the second message queue system.

[0138] According to one or more embodiments of the present disclosure, the first message queue system is deployed using a solid-state hard disk, and the second message queue system is deployed using a mechanical hard disk.

[0139] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0140] The memory stores computer-executable instructions;

[0141] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the data processing method based on the multi-message queue system as described in the first aspect and various possible designs of the first aspect.

[0142] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the data processing method based on a multi-message queue system as described in the first aspect and various possible designs of the first aspect is implemented.

[0143] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the data processing method based on a multi-message queue system as described in the first aspect and various possible designs of the first aspect.

[0144] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0145] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0146] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A data processing method based on a multi-message queue system, It is characterized in that include: receiving a data read request, wherein the data read request is used to request to read first target data; If the data reading request belongs to an instant service scenario, using a data access interface of a first message queue system in a heterogeneous multi-message queue system, read the first target data from the first message queue system for response; If the data read request belongs to an intensive service scenario, the data access interface of the second message queue system in the heterogeneous multi-message queue system is used to read the first target data from the second message queue system for response.

2. The data processing method according to claim 1, It is characterized in that The data read request carries a target service identifier, and the method further includes: A target service identifier is obtained from the data read request, and it is determined according to the target service identifier that the service scenario to which the data read request belongs is an instant service scenario or an intensive service scenario.

3. The data processing method according to claim 2, It is characterized in that If the target service identifier indicates an online service, determining that the data read request belongs to an instant service scenario; If the target service identifier indicates an offline service, it is determined that the data read request belongs to an intensive service scenario.

4. The data processing method according to claim 1, It is characterized in that The method further comprises: Receiving a data write request through a data access interface of a first message queue system in the heterogeneous multi-message queue system, wherein the data write request is used to request writing of second target data; Writing the second target data into the local storage of the first message queue system; The hierarchical storage management service is called to write the second target data into the local storage of the second message queue system based on a writing mode of writing the message queue at a specified offset.

5. The data processing method according to claim 1, It is characterized in that The method further comprises: Call the hierarchical storage management service to control the second message queue system to maintain data whose storage duration is greater than a preset time threshold, and delete the data whose storage duration is greater than the preset time threshold from the first message queue system; the second message queue system supports the first message queue system to access the full amount of data.

6. The data processing method according to claim 1, It is characterized in that The method of using a data access interface of a first message queue system in a heterogeneous multi-message queue system to read first target data from the first message queue system for response includes: Using the data access interface of the first message queue system in a heterogeneous multi-message queue system, adopting a local and remote-based data access strategy, first target data is read from the local storage of the first message queue system first, and when the data is insufficient, data is read from the local storage of the second message queue system and a response is made.

7. The data processing method according to claim 1, It is characterized in that The method of using a data access interface of a first message queue system in a heterogeneous multi-message queue system to read first target data from the first message queue system for response includes: Using the data access interface of the first message queue system in a heterogeneous multi-message queue system, adopting a data access strategy based on cold and hot data, first target data is read from the cache of the first message queue system first, and when the data is insufficient, data is read from the local storage of the first message queue system in response; wherein the cache of the first message queue system is used to store hot data.

8. The data processing method according to claim 1, It is characterized in that The first message queue system and the second message queue system adopt different hardware deployments; wherein the hard disk performance adopted by the first message queue system is higher than the hard disk performance adopted by the second message queue system.

9. The data processing method according to claim 7, It is characterized in that The first message queue system is deployed using a solid state drive, and the second message queue system is deployed using a mechanical hard drive.

10. A data processing device based on a multi-message queue system, It is characterized in that include: A receiving unit, configured to receive a data read request, wherein the data read request is configured to request to read first target data; A processing unit, configured to, if the data reading request belongs to an instant service scenario, use a data access interface of a first message queue system in a heterogeneous multi-message queue system to read first target data from the first message queue system for response; The processing unit is further used to, if the data reading request belongs to an intensive service scenario, use the data access interface of the second message queue system in the heterogeneous multi-message queue system to read the first target data from the second message queue system in response.

11. An electronic device, It is characterized in that include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the data processing method based on the multi-message queue system as described in any one of claims 1 to 9.

12. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the data processing method based on the multi-message queue system as described in any one of claims 1 to 9 is implemented.

13. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the data processing method based on a multi-message queue system as described in any one of claims 1 to 9 is implemented.

Citation Information

Patent Citations

  • System and method for preventing single-point bottleneck in a transactional middleware machine environment

    CN103827830A

  • Distributed storage system for NVMe and RDMA and lightweight synchronous communication method

    CN110262754A

  • Message storage method and system based on cold and hot separated storage

    CN114327280A

  • RDMA-based communication method, node, system and medium

    CN114691382A

  • Message processing method and device, electronic equipment and storage medium

    CN115866039A