Business processing system and business processing method

Abstract

The application provides a business processing system and a business processing method, which are applied to the technical field of financial technology, and the system comprises at least one client, a plurality of access agents connected with the client, a data storage partition of kafka connected with the access agent, and a distributed cache component, a target client receives a business processing request sent by a user, a first target access agent receives the business processing request, determines a target consumption thread based on a consumption state of the data storage partition and a consumption state of each consumption thread in the access agent, and then enables the target consumption thread to determine a target data storage partition and corresponding offset information from the data storage partition, the first target access agent determines corresponding target data based on the target data storage partition and the offset information, and returns the target data, an identifier of the target consumption thread, and an identifier of the target data storage partition to the target client. The technical scheme improves the message processing availability.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a business processing system and business processing method. Background Technology

[0002] Kafka, as a distributed streaming database, has gained increasing popularity in recent years. However, its structure is relatively complex, involving concepts such as topic, producer, consumer, and broker, which presents certain technical challenges in its use. Furthermore, direct connection to Kafka does not allow for control over client access permissions. Therefore, many Kafka server access proxies have emerged to address Kafka data consumption.

[0003] In existing technologies, common access proxy solutions solve the problems of connectivity and ease of access from the client to Kafka by encapsulating parameters and persisting long connections between the client and Kafka. That is, by encapsulating parameters and persisting long connections between the client and Kafka, the consumption partition of Kafka messages is determined when the consumer group is established. Because the access proxy records the partition information and message offset of Kafka, the client can only connect to a fixed access proxy for data consumption and message confirmation.

[0004] In the above method, the access proxy stores the storage partition information of the consumption and the data offset information in that partition. The client's consumption and submission must depend on the context information of the access proxy, which cannot guarantee the high availability requirement. Summary of the Invention

[0005] This application provides a business processing system and a business processing method to solve the problem of low availability of message processing in the prior art.

[0006] In a first aspect, embodiments of this application provide a business processing system, including: at least one client, multiple access proxies connected to the at least one client, multiple data storage partitions of Kafka connected to the multiple access proxies, and a distributed caching component;

[0007] The target client receives a service processing request from the user, and the target client is one of the at least one client;

[0008] The first target access proxy receives the business processing request, determines the target consumption thread based on the consumption status of the multiple data storage partitions in Kafka and the consumption status of each consumption thread in the multiple access proxies, and then activates the target consumption thread to determine the target data storage partition and corresponding offset information from the multiple data storage partitions. The first target access proxy is any one of the multiple access proxies determined based on the load balancing mechanism. The distributed cache component is used to record the consumption status of the multiple data storage partitions in Kafka and the consumption status of each consumption thread in the multiple access proxies.

[0009] The first target access proxy determines the corresponding target data based on the target data storage partition and offset information, and returns the target data, the identifier of the target consumption thread, and the identifier of the target data storage partition to the target client.

[0010] In one possible design of the first aspect, the system further includes: a second target access proxy, which is any one of the plurality of access proxies determined based on a load balancing mechanism;

[0011] The target client sends data confirmation information to the second target access proxy. The data confirmation information includes: the identifier of the target consumption thread and the identifier of the target data storage partition.

[0012] The second target access proxy detects whether the target consumer thread is holding a lock in the distributed caching component;

[0013] If the target consumer thread is locked, then release the lock on the target consumer thread and the lock on the target data storage partition.

[0014] Secondly, embodiments of this application provide a business processing method, the method being applied to a first target access proxy in a system according to any one of the first aspects and various possible designs, the method comprising:

[0015] Upon receiving a business processing request from the target client, the target consumption thread is activated to determine the target data storage partition and its corresponding offset information from the multiple data storage partitions;

[0016] Based on the target data storage partition and offset information, the corresponding target data is determined and returned to the target client.

[0017] In one possible design of the second aspect, before the target consumption thread is activated to determine the target data storage partition and its corresponding offset information from the plurality of data storage partitions, the method further includes:

[0018] In the distributed caching component, determine whether any consumer thread has an idle state.

[0019] If there are consumer threads in an idle state, determine any one of the idle consumer threads that is not locked, and take the lockless consumer thread as the target consumer thread. The lockless consumer thread indicates that the consumer thread has not supplied data.

[0020] Optionally, the method further includes:

[0021] If there are no consumer threads in an idle state, determine the total number of consumer threads and the total number of data storage partitions;

[0022] If the total number of consumer threads is less than the total number of data storage partitions, create a new consumer thread connected to the Kafka, and use the new consumer thread as the target consumer thread.

[0023] Optionally, the step of enabling the target consumption thread to determine the target data storage partition and its corresponding offset information from the plurality of data storage partitions includes:

[0024] The target consumer thread is locked and synchronized to the distributed cache component;

[0025] In the distributed caching component, a first data storage partition that matches the identifier of the service in the service processing request is identified;

[0026] If the consumption status of the first data storage partition is not locked, then the first data storage partition is determined as the target data storage partition;

[0027] The target data storage partition is locked and synchronized to the distributed cache component, and the corresponding offset information is determined.

[0028] Optionally, determining the first data storage partition in the distributed caching component that matches the identifier of the service in the service processing request includes:

[0029] Based on the identifiers of each data storage partition in the distributed caching component, detect the data storage partition that matches the identifier of the business in the business processing request;

[0030] The data storage partition that matches the identifier of the service in the service processing request shall be designated as the first data storage partition.

[0031] Optionally, after determining the corresponding target data, the method further includes:

[0032] The identifier of the target consumer thread and the identifier of the target data storage partition are sent to the target client.

[0033] Thirdly, embodiments of this application provide a business processing method, the method being applied to a second target access proxy in a system according to any one of the first aspects and various possible designs, the method comprising:

[0034] Receive data confirmation information sent by the target client, the data confirmation information including: the identifier of the target consumption thread and the identifier of the target data storage partition;

[0035] In the distributed caching component, detect whether the target consumer thread is holding a lock;

[0036] If the target consumer thread is locked, then release the lock on the target consumer thread and the lock on the target data storage partition.

[0037] Fourthly, embodiments of this application provide a service processing apparatus applied to a first target access proxy, the apparatus comprising:

[0038] The processing module is used to receive business processing requests sent by the target client and enable the target consumption thread to determine the target data storage partition and its corresponding offset information from the multiple data storage partitions;

[0039] The determination module is used to determine the corresponding target data based on the target data storage partition and offset information, and return the target data to the target client.

[0040] In one possible design of the fourth aspect, before the target consumption thread is activated to determine the target data storage partition and its corresponding offset information from the plurality of data storage partitions, the determining module is further configured to:

[0041] In the distributed caching component, determine whether any consumer thread has an idle state.

[0042] If there are consumer threads in an idle state, determine any one of the idle consumer threads that is not locked, and take the lockless consumer thread as the target consumer thread. The lockless consumer thread indicates that the consumer thread has not supplied data.

[0043] Optionally, the determining module is further configured to:

[0044] If there are no consumer threads in an idle state, determine the total number of consumer threads and the total number of data storage partitions;

[0045] If the total number of consumer threads is less than the total number of data storage partitions, create a new consumer thread connected to the Kafka, and use the new consumer thread as the target consumer thread.

[0046] Optionally, the processing module enables the target consumer thread to determine the target data storage partition and its corresponding offset information from the plurality of data storage partitions, specifically for:

[0047] The target consumer thread is locked and synchronized to the distributed cache component;

[0048] In the distributed caching component, a first data storage partition that matches the identifier of the service in the service processing request is identified;

[0049] If the consumption status of the first data storage partition is not locked, then the first data storage partition is determined as the target data storage partition;

[0050] The target data storage partition is locked and synchronized to the distributed cache component, and the corresponding offset information is determined.

[0051] Optionally, the determining module determines a first data storage partition in the distributed caching component that matches the identifier of the service in the service processing request, specifically for:

[0052] Based on the identifiers of each data storage partition in the distributed caching component, detect the data storage partition that matches the identifier of the business in the business processing request;

[0053] The data storage partition that matches the identifier of the service in the service processing request shall be designated as the first data storage partition.

[0054] Optionally, after the corresponding target data is determined, the sending module is configured to:

[0055] The identifier of the target consumer thread and the identifier of the target data storage partition are sent to the target client.

[0056] Fifthly, embodiments of this application provide a service processing apparatus applied to a second target access proxy, the apparatus comprising:

[0057] The receiving module is used to receive data confirmation information sent by the target client. The data confirmation information includes: the identifier of the target consumption thread and the identifier of the target data storage partition.

[0058] The processing module is used to detect whether the target consumer thread is locked in the distributed caching component, and when the target consumer thread is locked, to release the lock of the target consumer thread and the lock of the target data storage partition.

[0059] In a sixth aspect, this application provides an electronic device, including: a processor, and a memory and a transceiver communicatively connected to the processor;

[0060] The memory stores computer-executed instructions; the transceiver is used for sending and receiving data.

[0061] The processor executes computer execution instructions stored in the memory to implement the business processing method as described in the second or third aspect or any of the above.

[0062] In a seventh aspect, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the business processing methods described in the second or third aspects or any of the above-mentioned methods.

[0063] Eighthly, this application provides a computer program product, which, when executed by a processor, is used to implement the business processing method described in the second or third aspects or any of the above methods.

[0064] The business processing system and method provided in this application include: at least one client, multiple access proxies connected to the at least one client, multiple data storage partitions of Kafka connected to the multiple access proxies, and a distributed caching component. A target client receives a business processing request from a user. The target client is one of the at least one client. A first target access proxy receives the business processing request and, based on the consumption status of the multiple data storage partitions in Kafka and the consumption status of each consumption thread in the multiple access proxies, determines a target consumption thread. Then, the target consumption thread is activated to determine the target data storage partition and its corresponding offset information from the multiple data storage partitions. The first target access proxy is any one of the multiple access proxies determined based on a load balancing mechanism. The distributed caching component records the consumption status of the multiple data storage partitions in Kafka and the consumption status of each consumption thread in the multiple access proxies. Based on the target data storage partition and offset information, the first target access proxy determines the corresponding target data and returns the target data, the identifier of the target consumption thread, and the identifier of the target data storage partition to the target client. This technical solution ensures the decoupling of the proxy layer and the client by making the access proxy service information stateless. This prevents the client's data consumption from being affected by the stability of the Kafka database and the access proxy itself, thereby improving message processing availability, as well as the resource utilization and concurrent consumption capabilities of the access proxy. Attached Figure Description

[0065] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0066] Figure 1 A schematic diagram of a prior art business processing system provided for embodiments of this application;

[0067] Figure 2A Schematic diagram of the business processing system provided in the embodiments of this application Figure 1 ;

[0068] Figure 2B Schematic diagram 2 of the business processing system provided in the embodiments of this application;

[0069] Figure 3 A flowchart illustrating an embodiment of the business processing method provided in this application;

[0070] Figure 4 A flowchart illustrating Embodiment 2 of the business processing method provided in this application;

[0071] Figure 5 A flowchart illustrating Embodiment 3 of the business processing method provided in this application;

[0072] Figure 6 A flowchart illustrating Embodiment 4 of the business processing method provided in this application;

[0073] Figure 7 A flowchart illustrating Embodiment 5 of the business processing method provided in this application;

[0074] Figure 8 A flowchart illustrating Embodiment Six of the business processing method provided in this application;

[0075] Figure 9 Schematic diagram of the business processing apparatus provided in the embodiments of this application Figure 1 ;

[0076] Figure 10 Schematic diagram 2 of the business processing device provided in the embodiments of this application;

[0077] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0078] The accompanying drawings have illustrated specific embodiments of this disclosure, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this disclosure to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0080] Before introducing the embodiments of this application, the technical terms and application background of the embodiments of this application will be explained first:

[0081] Kafka: A distributed streaming platform widely used for its high throughput, persistence, horizontal scalability, and support for streaming data processing.

[0082] Streaming data is a series of data that arrives sequentially, in large quantities, rapidly, and continuously. Generally, streaming data can be viewed as a dynamic data set that grows infinitely over time.

[0083] Eventually consistent: All data replicas in a system eventually reach a consistent state after a certain period of time, without requiring real-time guarantees of strong consistency. Eventually consistent is a special case of weak consistency. The BASE theory is geared towards large-scale, highly available, and scalable distributed systems, sacrificing strong consistency for high availability.

[0084] Stateless service: A service that processes a single request without depending on other requests. In other words, all the information needed to process a request is either contained within the request itself or can be obtained from an external source; the server itself does not store any information.

[0085] Stateful service: It stores some data on its own, and the requests from different times are related.

[0086] ACID: Four properties that a database management system must possess to ensure the correctness and reliability of transactions during the writing or updating process: Atomicity, Consistency, Isolation, and Durability.

[0087] Kafka, as a distributed streaming database, has gained increasing popularity in recent years. However, its structure is relatively complex, involving concepts such as topic, producer, consumer, and broker, which presents certain technical challenges in its use. Furthermore, direct connection to Kafka does not allow for control over client access permissions. Therefore, many Kafka server access proxies have emerged to address Kafka data consumption.

[0088] Currently, common access proxy solutions solve the connectivity and ease of access from the client to Kafka by encapsulating parameters and persisting long-lived connections between the client and Kafka. However, since Kafka is a distributed streaming database, it does not provide the strong consistency concept of ACID. Furthermore, due to the inherent high availability of distributed databases, Kafka consumer rebalancing operations are triggered when the system or consumer experiences anomalies. Therefore, current access proxy solutions cannot guarantee consumer consistency, potentially leading to inconsistencies or duplicate consumption. Additionally, because access proxies consume data via long-lived connections, they store the partition (data storage partition) and offset (offset information) of the consumed Kafka messages on the access proxy itself.

[0089] Specifically, Figure 1 A schematic diagram of a prior art business processing system provided in the embodiments of this application, such as... Figure 1 As shown, the business processing system includes: client 11, client 12, and client 13; access proxy 14, access proxy 15, and access proxy 16; and Kafka server 17.

[0090] Among them, Kafka server 17 includes: data storage partition 01, data storage partition 02, data storage partition 03, data storage partition 04, data storage partition 05, data storage partition 06, data storage partition 07, data storage partition 08, and data storage partition 09.

[0091] As above Figure 1As shown, a common access proxy solution currently uses one or more access proxies to form a consumer group to consume Kafka data. The client data request component establishes a long-lived connection with Kafka through parameter encapsulation and persistence. The Kafka message consumption partition is determined when the consumer group is established because the access proxy records the Kafka partition information and message offset. Clients can only connect to a fixed access proxy for data consumption and message confirmation. This relatively fixed access pattern reduces the difficulty of using Kafka and allows for client authentication at the access proxy level, preventing information leakage. However, current solutions have several problems:

[0092] 1. The access proxy stores the consumption partition and offset. The client's consumption and submission must rely on the context information of the access proxy, which cannot guarantee the high availability requirement.

[0093] 2. When Kafka or the access broker experiences a system anomaly, it triggers Kafka's rebalancing mechanism, causing the partition consumed by the access broker to change. In this case, the data consumed by the client cannot be committed in a timely manner, resulting in duplicate consumption of data.

[0094] 3. When the volume of business data increases, there is no way to dynamically expand according to the actual volume of business data. It is necessary to shut down the system to redistribute the consumer groups, resulting in a relatively low data consumption capacity.

[0095] To address the technical problems existing in the prior art, the inventors of this application propose the following: If the consumption status of multiple data storage partitions of Kafka and the consumption status of each consumption thread in multiple access proxies are stored in a distributed cache component, when the client receives a business processing request, the access proxies that handle the business processing request are assigned based on the information recorded in the distributed cache component, and the consumption thread and corresponding data storage partition are determined. This decouples the client and the access proxies, thereby realizing the stateless scaling capability of the access proxies in large business access scenarios, and improving the overall stability and high availability of the system.

[0096] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0097] The technical solution of this application will now be described in detail through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0098] It is worth noting that the application areas of this publicly disclosed business processing system and business processing methods are not limited, and can include financial technology, the Internet, etc.

[0099] Figure 2A Schematic diagram of the business processing system provided in the embodiments of this application Figure 1 ,like Figure 2A As shown, the business processing system may include: at least one client 21, multiple access brokers 22 connected to at least one client 21, multiple data storage partitions 23 of Kafka connected to the multiple access brokers 22, and a distributed cache component 25.

[0100] The distributed cache component 25 is used to record the consumption status of multiple data storage partitions 23 in Kafka and the consumption status of each consumer thread in multiple access brokers 22; the distributed cache component 25 can be a Redis cluster.

[0101] One possible implementation is illustrated using two clients 21 (211, 212), four access proxies 22 (221, 222, 223, 224), and ten data storage partitions 23 (231, 232…239, 230).

[0102] It should be understood that for each access agent 22, there is at least one consumer thread.

[0103] Optionally, the target client 211 receives the service processing request sent by the user, and the target client 211 is one of at least one client 21.

[0104] The first target access proxy 221 receives the business processing request and determines the target consumer thread 24 based on the consumption status of multiple data storage partitions 23 in Kafka and the consumption status of each consumer thread in multiple access proxies 22. Then, the target consumer thread 24 is activated to determine the target data storage partition and the corresponding offset information from the multiple data storage partitions 23. The first target access proxy 221 is any one of the multiple access proxies 21 determined based on the load balancing mechanism.

[0105] The first target access proxy 221 determines the corresponding target data based on the target data storage partition 231 and offset information, and returns the target data, the identifier of the target consumer thread 24, and the identifier of the target data storage partition 231 to the target client 211.

[0106] Furthermore, Figure 2B The second schematic diagram of the architecture of the business processing system provided in the embodiments of this application is as follows: Figure 2B As shown, the business processing system also includes: a second target access agent 224.

[0107] The target client 211 sends data confirmation information to the second target access proxy 224. The data confirmation information includes: the identifier of the target consumer thread 24 and the identifier of the target data storage partition 231. The second target access proxy 224 is any one of the multiple access proxies 22 determined based on the load balancing mechanism.

[0108] Optionally, the second target access proxy 224 and the first target access proxy 221 can be the same access proxy, both determined by the load balancing mechanism.

[0109] In one possible implementation, after receiving a business processing request from a user, the load balancing mechanism in the business processing system is triggered, and the first target access proxy 221 is selected as the component to process the business processing request. Based on the information in the distributed caching component 25, the first target access proxy 221 determines the consumer thread that can be used to process the business processing request, namely the target consumer thread 24. Based on the information in the distributed caching component 25, the target consumer thread 24 determines the target data storage partition 231 and determines the target data corresponding to the business processing request from the target data storage partition 231. Then, the target data, the identifier of the target data storage partition 231, and the identifier of the target consumer thread 24 are returned to the target client 211 to be sent to the user.

[0110] Furthermore, after receiving the above information, the target client 211 sends the data confirmation information to the second target access agent 224 selected based on the load balancing mechanism, so as to notify the business processing system that the response to the business processing request has ended.

[0111] It should be understood that this application does not limit the number of clients, access proxies, or data storage partitions, and the following method embodiments cover any content not detailed or disclosed above.

[0112] The business processing system provided in this application includes: at least one client, multiple access proxies connected to the at least one client, multiple data storage partitions of Kafka connected to the multiple access proxies, and a distributed caching component. A target client receives a business processing request from a user. The target client is one of the at least one client. A first target access proxy receives the business processing request and, based on the consumption status of the multiple data storage partitions in Kafka and the consumption status of each consumption thread in the multiple access proxies, determines a target consumption thread. Then, the target consumption thread is activated to determine the target data storage partition and its corresponding offset information from the multiple data storage partitions. The first target access proxy is any one of the multiple access proxies determined based on a load balancing mechanism. The distributed caching component records the consumption status of the multiple data storage partitions in Kafka and the consumption status of each consumption thread in the multiple access proxies. Based on the target data storage partition and offset information, the first target access proxy determines the corresponding target data and returns the target data, the identifier of the target consumption thread, and the identifier of the target data storage partition to the target client. This technical solution ensures the decoupling of the proxy layer and the client by making the access proxy service information stateless. This prevents the client's data consumption from being affected by the stability of the Kafka database and the access proxy itself, thereby improving message processing availability, as well as the resource utilization and concurrent consumption capabilities of the access proxy.

[0113] Based on the above-described business processing system implementation examples Figure 3 This is a flowchart illustrating an embodiment of the business processing method provided in this application. Figure 3 The explanation will focus on the primary target access proxy as the executing entity, such as... Figure 3 As shown, the business processing method includes the following steps:

[0114] Step 31: Receive the business processing request sent by the target client, and enable the target consumer thread to determine the target data storage partition and its corresponding offset information from multiple data storage partitions.

[0115] The target client is one of multiple clients in the business processing system. In response to a business processing request sent by a user, the target client forwards the business processing request to the first target access proxy after receiving the request.

[0116] The first target access proxy is any one of the multiple access proxies in the business processing system determined based on the load balancing mechanism. The load balancing mechanism is used to achieve high availability. Common load balancing software includes Nginx, LVS, and hardware F5.

[0117] In this step, after the first target access proxy receives the business processing request, it starts the target consumption thread in its own consumption thread (the determination of the target consumption thread is given in the following embodiment). The target consumption thread consumes the business processing request, that is, it determines the target data storage partition and the corresponding offset information from multiple data storage partitions (the determination of the target data storage partition and the corresponding offset information is given in the following embodiment).

[0118] The target data storage partition and its corresponding offset information record the location where the target data corresponding to the business processing request is stored.

[0119] It should be understood that: a partition is any physical partition in the Kafka server where data is stored, and the offset records the address of each piece of data in that partition, i.e., the starting position of the cache; the implementation of the first target access proxy can be a poll logic, where the target client connects to the access proxy service to consume data based on the authentication information of the access proxy service.

[0120] Step 32: Based on the target data storage partition and offset information, determine the corresponding target data and return the target data to the target client.

[0121] In this step, after determining the target data storage partition and offset information as described above, the target data corresponding to the business processing request can be determined based on the corresponding offset in the target data storage partition. Then, the data is extracted and returned to the target client.

[0122] In addition, before pulling the target data, you can first submit the offset of the last consumption to the corresponding Kafka partition list, and then start pulling and consuming the target data and returning it to the client.

[0123] It should be understood that when returning the target data to the target client, the identifier of the target consumer thread and the identifier of the target data storage partition can also be carried for subsequent confirmation that the business processing request has been completed.

[0124] The business processing method provided in this application embodiment receives a business processing request from a target client, activates a target consumption thread to determine the target data storage partition and its corresponding offset information from multiple data storage partitions, determines the corresponding target data based on the target data storage partition and offset information, and returns the target data to the target client. This technical solution uses a target consumption thread to determine the target data storage partition and its corresponding offset information from multiple data storage partitions, thereby achieving data acquisition.

[0125] Based on the above embodiments, Figure 4The flowchart of Embodiment 2 of the business processing method provided in this application is shown below. Figure 4 As shown, prior to step 31, the business processing method further includes the following steps:

[0126] Step 41: In the distributed caching component, determine whether there are any consumer threads whose consumption status is idle.

[0127] In this scheme, before activating the target consumer thread, the target consumer thread needs to be determined first. After the business processing request is assigned to the first target access proxy for processing, the first target access proxy determines the consumer thread that will handle the business processing request from among its multiple consumer threads, i.e., the target consumer thread.

[0128] In this step, the distributed cache component stores the consumption status of each consumer thread in each access proxy. At this time, the first target access proxy determines in the distributed cache component whether there is a consumer thread with an idle consumption status among its multiple consumer threads.

[0129] Step 42: If there are consumer threads in an idle state, determine any one of the idle consumer threads that is not locked, and take any one of the unlocked consumer threads as the target consumer thread. The unlocked consumer thread indicates that the consumer thread has not supplied data.

[0130] In this step, if there are consumer threads in an idle state, first identify at least one consumer thread in the idle state, then identify the unlocked consumer thread among the at least one idle consumer thread, and use any one of the unlocked consumer threads as the target consumer thread to process business processing requests.

[0131] Locking a consumer thread indicates that the consumer thread is supplying data, thus preventing duplicate consumption. One consumer thread corresponds to one consumer thread lock (i.e., one consumer-key corresponds to one consumer).

[0132] Step 43: If there are no consumer threads in an idle state, determine the total number of consumer threads and the total number of data storage partitions.

[0133] In this step, if there are no consumer threads in an idle state, it means that all consumer threads are currently occupied. At this time, query the distributed cache component for the total number of consumer threads and the total number of data storage partitions.

[0134] Step 44: If the total number of consumer threads is less than the total number of data storage partitions, create a new consumer thread connected to Kafka and set the new consumer thread as the target consumer thread.

[0135] In this step, if the total number of consumer threads is less than the total number of data storage partitions, a new consumer thread can be created to connect to Kafka. This involves creating a new consumer and adding it to the connection pool, then using this new consumer thread as the target consumer thread.

[0136] In addition, the consumer-number in the distributed cache component can be incremented, which means adding information about new consumer threads to the distributed cache component.

[0137] The business processing method provided in this application's embodiments determines whether any consumer threads in the distributed caching component are in an idle state. If idle consumer threads exist, any unlocked consumer thread among these idle threads is selected and designated as the target consumer thread. An unlocked consumer thread indicates that it is not supplying data. If no idle consumer threads exist, the total number of consumer threads and the total number of data storage partitions are determined. If the total number of consumer threads is less than the total number of data storage partitions, a new consumer thread connected to Kafka is created and designated as the target consumer thread. This technical solution effectively determines the target consumer thread, providing a foundation for subsequent data acquisition.

[0138] Based on the above embodiments, Figure 5 The flowchart of Embodiment 3 of the business processing method provided in this application is shown below. Figure 5 As shown, step 31 may include the following steps:

[0139] Step 51: Lock the target consumer thread and synchronize it to the distributed cache component;

[0140] In this step, after identifying the target consumer thread, the target consumer thread is locked, and the locking information is synchronized to the distributed cache component.

[0141] This step is implemented so that when the access proxy performs processing in subsequent visits, it can continue to support the client's data consumption based on this information, without causing the client to consume data repeatedly.

[0142] Step 52: In the distributed cache component, identify the first data storage partition that matches the business identifier in the business processing request.

[0143] Optionally, step 52 can be implemented as follows: based on the identifiers of each data storage partition in the distributed cache component, detect the data storage partition that matches the identifier of the business in the business processing request; and use the data storage partition that matches the identifier of the business in the business processing request as the first data storage partition.

[0144] Step 53: If the consumption state of the first data storage partition is not locked, then the first data storage partition is determined as the target data storage partition;

[0145] In this step, if the consumption state of the first data storage partition is not locked, it means that the first data storage partition is not currently in the data supply stage. At this time, the first data storage partition is determined as the target data storage partition.

[0146] Furthermore, if the consumption status of the first data storage partition is locked, it means that the consumption offset of the first data storage partition has not been updated and consumption is not allowed.

[0147] Step 54: Lock the target data storage partition, synchronize it to the distributed cache component, and determine the corresponding offset information.

[0148] In this step, the first target access proxy locks the target data storage partition and synchronizes the locking information to the distributed cache component. The reason for this implementation is the same as in step 51.

[0149] The determination of the corresponding offset information is based on the implementation of step 52.

[0150] The business processing method provided in this application locks the target consumption thread and synchronizes it to a distributed cache component. The distributed cache component identifies a first data storage partition that matches the business identifier in the business processing request. If the consumption state of the first data storage partition is not locked, it is designated as the target data storage partition, locked, synchronized to the distributed cache component, and the corresponding offset information is determined. This technical solution locks both the data storage partition processing the business processing request and the target consumption thread to prevent duplicate data consumption.

[0151] Based on the above embodiments, Figure 6 This is a flowchart illustrating Embodiment 4 of the business processing method provided in this application. Figure 6 The explanation will focus on the second target access proxy as the execution entity, such as... Figure 6 As shown, the business processing method includes the following steps:

[0152] Step 61: Receive data confirmation information sent by the target client. The data confirmation information includes: the identifier of the target consumption thread and the identifier of the target data storage partition.

[0153] In this scheme, after the first target access proxy returns the target data to the target client, it is also necessary to notify the business processing system that the business processing request has ended. At this time, the second target access proxy receives the data confirmation information sent by the target client.

[0154] It should be understood that the second target access proxy is also determined from multiple access proxies in the business processing system based on the load balancing mechanism. There is a possibility that the second target access proxy is the same access proxy as the first target access proxy.

[0155] The data confirmation information can be a commitSync request.

[0156] Step 62: In the distributed caching component, check whether the target consumer thread is holding a lock;

[0157] In this step, the first target access proxy synchronizes the locking information of the target consumer thread to the distributed cache component in the above steps. At this time, the second target access proxy determines whether the target consumer thread has locked (i.e., whether the consumer-key exists) in the distributed cache component based on the identifier of the target consumer thread.

[0158] Step 63: If the target consumer thread is holding a lock, release the lock on the target consumer thread and the lock on the target data storage partition.

[0159] In this step, if the target consumer thread is locked, the lock of the target consumer thread and the lock of the target data storage partition are released, that is, the corresponding partition-key is released, and finally the consumer-key is released to complete a complete data consumption.

[0160] In addition, the offset information in the distributed cache is updated.

[0161] If the target consumer thread does not have a lock, no action is taken to prevent accidental submissions and data loss.

[0162] The business processing method provided in this application embodiment receives data confirmation information sent by a target client. The data confirmation information includes: the identifier of the target consumer thread and the identifier of the target data storage partition. The method detects whether the target consumer thread is holding a lock in a distributed caching component. If the target consumer thread is holding a lock, the lock on the target consumer thread and the lock on the target data storage partition are released. This technical solution sends data confirmation information to the target client after data extraction to unlock the lock on the target consumer thread and the target data storage partition.

[0163] also, Figure 7 This is a flowchart illustrating Embodiment 5 of the business processing method provided in this application, explaining the data acquisition (i.e., accessing proxy data poll (i.e., pulling data) logic). The specific execution order is shown below. Figure 7 .

[0164] Step 1: Obtain the consumption status of each consumer thread;

[0165] Step 2: Determine if there are any consumer threads in an idle state;

[0166] Step 3: If it does not exist, create a new consumer thread (the total number of consumer threads is less than the total number of data storage partitions) as the target consumer thread, and execute step 5.

[0167] Step 4: If it exists, identify an unlocked consumer thread as the target consumer thread;

[0168] Step 5: Lock the target consumer thread;

[0169] Step 6: Obtain the consumption status of each data storage partition;

[0170] Step 7: Determine if there is a data storage partition whose consumption status is unlocked;

[0171] Step 8: Based on the business processing request, determine the target data storage partition in the unlocked data storage partition;

[0172] Step 9: Lock the target data storage partition;

[0173] Step 10: Retrieve the target data from the target data storage partition and return the identifier of the target data storage partition and the identifier of the target consumption thread to the target client.

[0174] Furthermore, Figure 8 This is a flowchart illustrating Embodiment Six of the business processing method provided in this application, explaining the confirmation and retrieval process after data acquisition (i.e., the commitSync logic for accessing proxy data). The specific execution order is shown below. Figure 8 .

[0175] Step 1: Receive data confirmation information sent by the target client;

[0176] Step 2: Based on the identifier of the target consumer thread in the data confirmation information, determine whether the target consumer thread has acquired a lock;

[0177] Step 3: If locks exist, release the locks of the target consumer thread and the target data storage partition.

[0178] That is, based on the above technical solution, the embodiments of this application can achieve the following:

[0179] Firstly, by introducing distributed locks and distributed caching, the partition and offset information of the access proxy is stored in a distributed caching component (such as Redis), which realizes the statelessness of the access proxy and the decoupling between the client and the access proxy. Therefore, each consumption and submission of the client does not need to be completed with the same access proxy, but can be randomly sent to any access proxy through load balancing. Thus, the failure of any access proxy will not affect the client's data consumption.

[0180] Secondly, when a Kafka or access broker malfunctions and triggers a rebalancing, the access broker can continue to support client data consumption based on the cached information since the distributed caching component stores the consumption status of the Kafka cluster, thus preventing duplicate data consumption by the client.

[0181] Thirdly, due to the stateless nature of the access proxy layer, when the volume of business data surges or shrinks, the access proxy can be dynamically scaled up or down by simply adding or removing access proxies, thereby improving resource utilization and the processing capability of big data business data.

[0182] Fourthly, when Kafka encounters an anomaly, the broker malfunctions, or the broker dynamically expands, the change in the number of consumers will trigger a Kafka rebalancing. During this time, the number of partitions within the consumers of the service broker will change. However, because Kafka consumer state information is stored in a distributed cache, neither client pull requests nor commitSync requests will be affected by the rebalancing.

[0183] The following are embodiments of the business processing apparatus of this application, which can be used to execute embodiments of the business processing method of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of this application.

[0184] Figure 9 Schematic diagram of the business processing apparatus provided in the embodiments of this application Figure 1 .like Figure 9 As shown, the service processing apparatus is applied to a first target access proxy, and the service processing apparatus includes:

[0185] Processing module 91 is used to receive business processing requests sent by the target client and enable the target consumption thread to determine the target data storage partition and corresponding offset information from multiple data storage partitions;

[0186] The determination module 92 is used to determine the corresponding target data based on the target data storage partition and offset information, and return the target data to the target client.

[0187] In one possible design of this application embodiment, before the target consumer thread is enabled to determine the target data storage partition and its corresponding offset information from multiple data storage partitions, the determining module 92 is further configured to:

[0188] In the distributed caching component, determine whether any consumer thread has an idle state.

[0189] If there are consumer threads in an idle state, identify any one of the idle consumer threads that is not locked, and use that one as the target consumer thread. An unlocked consumer thread indicates that the consumer thread has not supplied data.

[0190] Optionally, module 92 is also used for:

[0191] If there are no consumer threads in an idle state, determine the total number of consumer threads and the total number of data storage partitions;

[0192] If the total number of consumer threads is less than the total number of data storage partitions, create a new consumer thread that connects to Kafka and set the new consumer thread as the target consumer thread.

[0193] Optionally, processing module 91 enables the target consumer thread to determine the target data storage partition and its corresponding offset information from multiple data storage partitions, specifically for:

[0194] Lock the target consumer thread and synchronize it to the distributed cache component;

[0195] In the distributed caching component, the first data storage partition that matches the business identifier in the business processing request is identified;

[0196] If the consumption state of the first data storage partition is not locked, then the first data storage partition is determined as the target data storage partition;

[0197] The target data storage partition is locked and synchronized to the distributed cache component, and the corresponding offset information is determined.

[0198] Optionally, the determining module 92 identifies a first data storage partition in the distributed caching component that matches the business identifier in the business processing request, specifically for:

[0199] Based on the identifiers of each data storage partition in the distributed caching component, detect the data storage partition that matches the identifier in the business processing request;

[0200] The data storage partition that matches the business identifier in the business processing request will be designated as the first data storage partition.

[0201] Optionally, after the corresponding target data is determined, the sending module is configured to:

[0202] Send the identifier of the target consumer thread and the identifier of the target data storage partition to the target client.

[0203] The service processing apparatus provided in this application embodiment can be used to execute the service processing determination method applied to any of the embodiments of the first target access agent. Its implementation principle and technical effect are similar, and will not be described again here.

[0204] Figure 10 A second schematic diagram of the structure of the service processing device provided in an embodiment of this application is shown. Figure 10 As shown, the service processing apparatus is applied to a second target access proxy, and the service processing apparatus includes:

[0205] The receiving module 101 is used to receive data confirmation information sent by the target client. The data confirmation information includes: the identifier of the target consumption thread and the identifier of the target data storage partition.

[0206] The processing module 102 is used to detect whether the target consumer thread has acquired a lock in the distributed cache component, and when the target consumer thread has acquired a lock, to release the lock of the target consumer thread and the lock of the target data storage partition.

[0207] The service processing apparatus provided in this application embodiment can be used to execute the service processing determination method applied to any of the embodiments of the second target access agent. Its implementation principle and technical effect are similar, and will not be described again here.

[0208] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. Additionally, these modules can be fully or partially integrated together, or implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through the integrated logic circuits in the hardware of the processor element or through software instructions.

[0209] Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 11 As shown, the electronic device may include: a processor 111, a memory 112, and computer program instructions stored in the memory 112 and executable on the processor 111. When the processor 111 executes the computer program instructions, it implements the method provided in any of the foregoing embodiments.

[0210] Optionally, the various components of the electronic device can be connected via a system bus.

[0211] The memory 112 can be a separate storage unit or a storage unit integrated into the processor 111. The number of processors 111 can be one or more.

[0212] It should be understood that the processor 111 can be a Central Processing Unit (CPU), or other general-purpose processors 111, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor 111 can be a microprocessor 111, or any conventional processor 111. The steps of the method disclosed in this application can be directly manifested as being executed by the hardware processor 111, or being executed by a combination of hardware and software modules within the processor 111.

[0213] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. Memory 112 may include random access memory (RAM) 112, and may also include non-volatile memory (NVM) 112, such as at least one disk storage device 112.

[0214] All or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory 112. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory 112 (storage medium) includes: read-only memory 112 (ROM), RAM, flash memory 112, hard disk, solid-state hard disk, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0215] The electronic device provided in this application embodiment can be used to execute the business processing method provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0216] This application provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the above-described method.

[0217] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0218] Optionally, a readable storage medium can be coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Alternatively, the readable storage medium can be an integral part of the processor. Both the processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components within the device.

[0219] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and the at least one processor can implement the above-mentioned business processing method when executing the computer program.

[0220] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A business processing system, characterized in that, include: At least one client, multiple access proxies connected to the at least one client, multiple data storage partitions of Kafka connected to the multiple access proxies, and a distributed caching component; The target client receives a service processing request from the user, and the target client is one of the at least one client; The first target access proxy receives the business processing request, determines the target consumption thread based on the consumption status of the multiple data storage partitions in Kafka and the consumption status of each consumption thread in the multiple access proxies, and then activates the target consumption thread to determine the target data storage partition and corresponding offset information from the multiple data storage partitions. The first target access proxy is any one of the multiple access proxies determined based on the load balancing mechanism. The distributed cache component is used to record the consumption status of the multiple data storage partitions in Kafka and the consumption status of each consumption thread in the multiple access proxies. The first target access proxy determines the corresponding target data based on the target data storage partition and offset information, and returns the target data, the identifier of the target consumption thread, and the identifier of the target data storage partition to the target client.

2. The system according to claim 1, characterized in that, The system further includes: a second target access proxy, which is any one of the plurality of access proxies determined based on a load balancing mechanism; The target client sends data confirmation information to the second target access proxy. The data confirmation information includes: the identifier of the target consumption thread and the identifier of the target data storage partition. The second target access proxy detects whether the target consumer thread is holding a lock in the distributed caching component; If the target consumer thread is locked, then release the lock on the target consumer thread and the lock on the target data storage partition.

3. A business processing method, characterized in that, The method is applied to a first target access agent in the system according to any one of claims 1 or 2, the method comprising: Receive the business processing request sent by the target client, and enable the target consumer thread to determine the target data storage partition and its corresponding offset information from multiple data storage partitions; Based on the target data storage partition and offset information, the corresponding target data is determined and returned to the target client.

4. The method according to claim 3, characterized in that, Before the target consumption thread is activated to determine the target data storage partition and its corresponding offset information from the plurality of data storage partitions, the method further includes: In the distributed caching component, determine whether any consumer thread has an idle state. If there are consumer threads in an idle state, determine any one of the idle consumer threads that is not locked, and take the lockless consumer thread as the target consumer thread. The lockless consumer thread indicates that the consumer thread has not supplied data.

5. The method according to claim 4, characterized in that, The method further includes: If there are no consumer threads in an idle state, determine the total number of consumer threads and the total number of data storage partitions; If the total number of consumer threads is less than the total number of data storage partitions, create a new consumer thread connected to Kafka and use the new consumer thread as the target consumer thread.

6. The method according to claim 5, characterized in that, The activation of the target consumption thread determines the target data storage partition and its corresponding offset information from the plurality of data storage partitions, including: The target consumer thread is locked and synchronized to the distributed cache component; In the distributed caching component, a first data storage partition that matches the identifier of the service in the service processing request is identified; If the consumption status of the first data storage partition is not locked, then the first data storage partition is determined as the target data storage partition; The target data storage partition is locked and synchronized to the distributed cache component, and the corresponding offset information is determined.

7. The method according to claim 6, characterized in that, The step of determining the first data storage partition in the distributed caching component that matches the identifier of the service in the service processing request includes: Based on the identifiers of each data storage partition in the distributed caching component, detect the data storage partition that matches the identifier of the business in the business processing request; The data storage partition that matches the identifier of the service in the service processing request shall be designated as the first data storage partition.

8. The method according to claim 6, characterized in that, After determining the corresponding target data, the method further includes: The identifier of the target consumer thread and the identifier of the target data storage partition are sent to the target client.

9. A business processing method, characterized in that, The method is applied to a second target access agent in the system described in claim 2, and the method includes: Receive data confirmation information sent by the target client, the data confirmation information including: the identifier of the target consumption thread and the identifier of the target data storage partition; In the distributed caching component, detect whether the target consumer thread is holding a lock; If the target consumer thread is locked, then release the lock on the target consumer thread and the lock on the target data storage partition.

10. A business processing apparatus, characterized in that, A first target access agent applied in the system according to any one of claims 1 or 2, the apparatus comprising: The processing module is used to receive business processing requests sent by the target client and enable the target consumer thread to determine the target data storage partition and its corresponding offset information from multiple data storage partitions; The determination module is used to determine the corresponding target data based on the target data storage partition and offset information, and return the target data to the target client.

11. A business processing apparatus, characterized in that, A second target access agent applied to the system of claim 2, the device comprising: The receiving module is used to receive data confirmation information sent by the target client. The data confirmation information includes: the identifier of the target consumption thread and the identifier of the target data storage partition. The processing module is used to detect whether the target consumer thread is locked in the distributed caching component, and when the target consumer thread is locked, to release the lock of the target consumer thread and the lock of the target data storage partition.

12. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 3 to 9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 3 to 9.

14. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, is used to implement the method as described in any one of claims 3 to 9.

Images