Data processing method and device, computer device and storage medium

Abstract

The application relates to a data processing method and device, computer equipment, a storage medium and a computer program product, relates to the technical field of big data, and can be used in the field of financial technology or other fields. The method comprises the following steps: in response to a data transmission instruction, a plurality of first communication messages are read from a target distributed message queue according to message sorting parameters, and each first communication message read out is distributed to a local queue corresponding to each third-party system; for each local queue, a second communication message is read from the local queue according to data transmission parameters corresponding to the local queue, and the second communication message read out is sent to a third-party system corresponding to the local queue; the data transmission parameters corresponding to the local queue are determined according to the processing capacity parameters of the third-party system corresponding to the local queue. The method can improve the data processing efficiency.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data processing method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] In certain scenarios, bank operating systems need to send a large number of communication messages to various third-party systems they connect to. For example, when a bank operating system distributes benefits or virtual resources (such as shopping platform coupons) from a third-party platform, after a user receives these benefits through the bank's application (such as mobile banking), the bank operating system needs to send communication messages corresponding to each user to each third-party system, instructing them to process the information (such as recording the corresponding virtual resources for the user to use smoothly on the third-party platform). With the application of distributed technology in bank operating systems, current bank operating systems have high concurrency throughput capabilities.

[0003] In related technologies, a bank's operating system can store the communication messages corresponding to each user in a distributed message queue, and then read the communication messages from the distributed message queue at a certain data transmission speed and transmit them to the corresponding third-party system for processing.

[0004] However, due to the varying processing capabilities of different third-party systems, if the data transmission speed used by the bank's operating system exceeds the processing capacity of some third-party systems, it can easily lead to the collapse of those third-party systems, making it impossible to process data in a timely manner, resulting in low data processing efficiency. Summary of the Invention

[0005] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve data processing efficiency in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a data processing method. The method includes:

[0007] In response to the data transmission command, according to the message sorting parameters, multiple first communication messages are read from the target distributed message queue, and each read first communication message is allocated to the local queue corresponding to each third-party system;

[0008] For each local queue, a second communication message is read from the local queue according to the data transmission parameters corresponding to the local queue, and the read second communication message is sent to the third-party system corresponding to the local queue; the data transmission parameters corresponding to the local queue are determined according to the processing capability parameters of the third-party system corresponding to the local queue.

[0009] In one embodiment, the processing capacity parameters of the third-party system include the transaction processing volume per unit time, and the message sorting parameters include a first time window and the target read quantity within the first time window; the method further includes:

[0010] Obtain the transaction volume of each third-party system within a unit of time, and calculate the sum of the transaction volumes within each unit of time.

[0011] Based on the sum, the number of processing nodes in the distributed system, and the preset first time window, the target number of reads within the first time window is determined.

[0012] In one embodiment, the step of reading multiple first communication messages from the target distributed message queue according to message sorting parameters, and allocating each read first communication message to the corresponding local queue of each third-party system, includes:

[0013] If the current time reaches the start time of the current first time window, read the target number of first communication messages from the target distributed message queue;

[0014] Based on the feature information contained in each of the first communication messages, the third-party system corresponding to each of the first communication messages is determined, and based on the third-party system corresponding to each of the first communication messages, each of the first communication messages is allocated to the local queue corresponding to each third-party system.

[0015] In one embodiment, the processing capacity parameter of the third-party system includes the transaction processing volume per unit time, and the data transmission parameter includes a second time window and a target transmission quantity within the second time window; the method further includes:

[0016] For each local queue, obtain the transaction volume of the corresponding third-party system within a unit of time.

[0017] The transaction volume per unit time of the local queue is determined based on the transaction volume per unit time of the third-party system and the number of processing nodes of the distributed system.

[0018] Based on the transaction volume per unit time corresponding to the local queue and the preset second time window, the target transmission quantity within the second time window corresponding to the local queue is determined.

[0019] In one embodiment, the step of reading a second communication message from each local queue according to the data transmission parameters corresponding to the local queue, and sending the read second communication message to the third-party system corresponding to the local queue, includes:

[0020] For each local queue, when the current time reaches the start time of the current second time window corresponding to the local queue, the target number of second communication messages are read from the local queue, and the read second communication messages are sent to the third-party system corresponding to the local queue.

[0021] In one embodiment, the data transmission parameters include a second time window and a target number of transmissions within the second time window; the step of reading a second communication message from the local queue according to the data transmission parameters corresponding to the local queue, and sending the read second communication message to the third-party system corresponding to the local queue, includes:

[0022] For each local queue, if the current time reaches the start time of the current second time window corresponding to the local queue, it is determined whether there is an abnormal message in the abnormal message queue corresponding to the local queue; the abnormal message queue is used to store communication messages whose processing result is a processing abnormality.

[0023] If an exception message exists in the exception message queue, read the exception message from the exception message queue;

[0024] If the first number of abnormal messages read is not less than the target transmission number, the read abnormal messages will be sent to the third-party system corresponding to the local queue.

[0025] If the first quantity is less than the target transmission quantity, a second quantity is determined based on the first quantity and the target transmission quantity, and the second quantity of second communication messages is read from the local queue;

[0026] The retrieved exception message and second communication message are sent to the third-party system corresponding to the local queue.

[0027] In one embodiment, after sending the read second communication message to the third-party system corresponding to the local queue, the method further includes:

[0028] For each local queue, the processing result of each communication message is determined based on the feedback information from the third-party system corresponding to the local queue, and communication messages whose processing result is an exception are stored as exception messages in the exception message queue corresponding to the local queue.

[0029] In one embodiment, after determining the processing result of each communication message based on feedback information from the third-party system corresponding to the local queue, the method further includes:

[0030] The processing result is determined as the ratio of the number of abnormal communication messages to the total number of communication messages processed, and if the ratio is greater than a preset threshold, a first alarm message is output.

[0031] In one embodiment, the method further includes:

[0032] For each of the local queues, if the number of communication messages in the local queue exceeds a preset threshold, a second alarm message is output.

[0033] Secondly, this application also provides a data processing apparatus. The apparatus includes:

[0034] The allocation module is used to respond to data transmission instructions, read multiple first communication messages from the target distributed message queue according to message sorting parameters, and allocate each read first communication message to the local queue corresponding to each third-party system.

[0035] The sending module is used to read a second communication message from each local queue according to the data transmission parameters corresponding to the local queue, and send the read second communication message to the third-party system corresponding to the local queue; the data transmission parameters corresponding to the local queue are determined according to the processing capability parameters of the third-party system corresponding to the local queue.

[0036] In one embodiment, the processing capacity parameters of the third-party system include the transaction processing volume per unit time, and the message sorting parameters include a first time window and the target read quantity within the first time window; the device further includes:

[0037] The first acquisition module is used to acquire the transaction volume of each third-party system within a unit of time and calculate the sum of the transaction volume of each unit of time.

[0038] The first determining module is used to determine the target number of reads within the first time window based on the sum, the number of processing nodes in the distributed system, and a preset first time window.

[0039] In one embodiment, the allocation module is specifically used for:

[0040] When the current time reaches the start time of the current first time window, read the target number of first communication messages from the target distributed message queue; determine the third-party system corresponding to each first communication message based on the feature information contained in each first communication message, and allocate each first communication message to the local queue corresponding to each third-party system based on the third-party system corresponding to each first communication message.

[0041] In one embodiment, the processing capacity parameter of the third-party system includes the transaction processing volume per unit time, and the data transmission parameter includes a second time window and a target transmission quantity within the second time window; the device further includes:

[0042] The second acquisition module is used to acquire the transaction processing volume of the third-party system corresponding to each local queue within a unit of time for each local queue.

[0043] The second determining module is used to determine the transaction volume of the local queue within a unit of time based on the transaction volume of the third-party system within a unit of time and the number of processing nodes of the distributed system.

[0044] The third determining module is used to determine the target transmission quantity within the second time window corresponding to the local queue based on the transaction processing volume per unit time corresponding to the local queue and the preset second time window.

[0045] In one embodiment, the sending module is specifically used for:

[0046] For each local queue, when the current time reaches the start time of the current second time window corresponding to the local queue, the target number of second communication messages are read from the local queue, and the read second communication messages are sent to the third-party system corresponding to the local queue.

[0047] In one embodiment, the data transmission parameters include a second time window and the target number of transmissions within the second time window; the sending module is specifically used for:

[0048] For each local queue, if the current time reaches the start time of the current second time window corresponding to the local queue, it is determined whether there is an abnormal message in the abnormal message queue corresponding to the local queue; the abnormal message queue is used to store communication messages whose processing result is a processing exception; if there is an abnormal message in the abnormal message queue, the abnormal message is read from the abnormal message queue; if the first number of read abnormal messages is not less than the target transmission number, the read abnormal messages are sent to the third-party system corresponding to the local queue; if the first number is less than the target transmission number, a second number is determined based on the first number and the target transmission number, and the second number of second communication messages are read from the local queue; the read abnormal messages and second communication messages are sent to the third-party system corresponding to the local queue.

[0049] In one embodiment, the device further includes:

[0050] The storage module is used to determine the processing result of each communication message for each local queue based on the feedback information from the third-party system corresponding to the local queue, and to store communication messages whose processing result is a processing exception as exception messages in the exception message queue corresponding to the local queue.

[0051] In one embodiment, the device further includes:

[0052] The first alarm module is used to determine the ratio of the number of communication messages with abnormal processing results to the total number of communication messages processed, and outputs the first alarm information when the ratio is greater than a preset threshold.

[0053] In one embodiment, the device further includes:

[0054] The second alarm module is used to output a second alarm message for each of the local queues when the number of communication messages in the local queue exceeds a preset threshold.

[0055] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described in the first aspect.

[0056] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0057] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0058] The aforementioned data processing method, apparatus, computer equipment, storage medium, and computer program product read multiple communication messages from a distributed message queue storing communication messages based on message sorting parameters (which reflect message sorting speed), allocate them to local queues corresponding to each third-party system, and read a second communication message from each local queue based on data transmission parameters (which reflect data transmission speed) and send it to the corresponding third-party system for processing. The data transmission parameters for each local queue are determined based on the processing capacity parameters of the corresponding third-party system and are matched to the processing capacity of the third-party system. Therefore, this method can avoid system crashes caused by data transmission speeds exceeding the processing capacity of some third-party systems; that is, this method can improve system stability and overall data processing efficiency. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of a distributed system in an example.

[0060] Figure 2 This is a flowchart illustrating a data processing method in one embodiment;

[0061] Figure 3 This is a flowchart illustrating the data processing method in another embodiment;

[0062] Figure 4 This is a schematic diagram of a distributed system in another example;

[0063] Figure 5 This is a flowchart illustrating the data processing method in another embodiment;

[0064] Figure 6 This is a schematic diagram illustrating the process of sending the read message to a third-party system in one embodiment.

[0065] Figure 7 This is a structural block diagram of a data processing device in one embodiment;

[0066] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] First, before introducing the technical solutions of the embodiments of this application in detail, we will first introduce the technical background or technical evolution on which the embodiments of this application are based. In some scenarios, such as when a bank operating system issues rights or virtual resources (such as shopping platform coupons) to a third-party shopping platform, the bank operating system needs to send communication messages corresponding to each user to the third-party system, instructing the third-party system to process the information (such as recording the virtual resources corresponding to the user so that the user can use them smoothly on the third-party platform).

[0069] For activities such as claiming benefits or other events, there may be a surge in traffic during the event period. To improve the system's concurrent throughput and stability, distributed technology has been applied to the bank's operating system. That is, the bank's operating system can adopt a distributed architecture, which can contain multiple distributed nodes, such as... Figure 1The distributed architecture of the bank's operating system shown contains N distributed nodes (e.g., sub-service layers set1 to setN). The service access layer of the bank's operating system can, based on certain rules such as user region, customer group, and tags, use consistent hashing routing to route each user's rights claim requests or other transaction requests to different distributed nodes for processing. For users who successfully claim their rights, the distributed nodes can generate a corresponding communication message and send it to the third-party system associated with that right for further processing.

[0070] In related technologies, each distributed node of a bank's operating system can store the generated communication messages corresponding to each user in its own message queue. Each distributed node can read the communication messages from the message queue at a certain data transmission rate (a preset fixed value), call the interface of the third-party system corresponding to the communication message, and transmit the communication message to the corresponding third-party system for processing.

[0071] However, due to the varying processing capabilities of different third-party systems, if the data transmission speed used by the banking system exceeds the processing capacity of a particular third-party system, it can easily lead to a cascading failure of that system, preventing timely data processing and resulting in low data processing efficiency. Using a lower data transmission speed also results in low overall efficiency. Furthermore, given the high concurrency capabilities of the banking system and the strong processing capabilities of some third-party systems, a lower data transmission speed cannot fully utilize the resources of the more capable systems to leverage their advantages. Against this backdrop, the applicant, through long-term research and development and experimental verification, proposes the data processing method of this application. This method decouples the data transmission threads connecting to various third-party systems, allowing each thread to send pending communication messages to the third-party system at a data transmission speed matched to its processing capacity. This avoids system crashes caused by data transmission speeds exceeding the processing capacity of third-party systems, thus preventing a decline in overall data processing efficiency. Moreover, the data transmission speed can be reasonably set according to the processing capacity of the third-party system to fully utilize system resources and improve resource utilization. It should also be noted that the applicant has devoted considerable creative effort to discovering the technical problem of this application and the technical solutions described in the following embodiments.

[0072] The data processing method provided in this application embodiment can be applied to, for example... Figure 1The distributed system 102 shown can be a bank operation system or other systems. Distributed system 102 can contain multiple distributed nodes (as shown in the sub-service layers set1 to setN), and each distributed node can communicate with multiple third-party systems 104 via a network. It is understood that each distributed node can contain multiple processing nodes. Distributed system 102 and third-party systems 104 can be implemented using computer devices such as terminals, servers, and / or server clusters.

[0073] In one embodiment, such as Figure 2 As shown, a data processing method is provided, which can be applied to... Figure 1 Taking a distributed system as an example, the following steps are included:

[0074] Step 201: In response to the data transmission instruction, according to the message sorting parameters, read multiple first communication messages from the target distributed message queue, and allocate each read first communication message to the local queue corresponding to each third-party system.

[0075] In implementation, the service access layer of the distributed system can route user requests (such as rights claim requests or other requests) to various distributed nodes (sub-service layers) for processing. After processing the user's request, each distributed node can generate a communication message to be sent to the corresponding third-party system based on the processing result (such as successful rights claim). This communication message can be stored in a distributed message queue (such as a Kafka message queue). Responding to data transmission instructions, each distributed node can read multiple communication messages from the target distributed message queue storing the messages to be sent to the third party, based on message sorting parameters. It then determines the corresponding third-party system based on the characteristic information contained in each message (such as the identifier information of the target field), and allocates each communication message to the local queue corresponding to the third-party system. The message sorting parameters can reflect the message sorting speed, such as the number of communication messages read per unit time.

[0076] Step 202: For each local queue, read the second communication message from the local queue according to the data transmission parameters corresponding to the local queue, and send the read second communication message to the third-party system corresponding to the local queue.

[0077] In implementation, for each local queue, the distributed nodes of the distributed system can read communication messages from the local queue through message processing threads based on the data transmission parameters corresponding to that local queue, and then send the read communication messages to the third-party system corresponding to that local queue. The data transmission parameters corresponding to the local queue are determined based on the processing capacity parameters of the third-party system corresponding to the local queue, reflecting the data transmission speed to that third-party system. For example, the data transmission parameter can be the number of transactions processed per unit time (e.g., transactions per second, tps). Reading a communication message from the local queue, sending the communication message to the third-party system, and determining the processing result of the communication message based on feedback information from the third-party system (if any) can be considered as a single transaction.

[0078] In the aforementioned data processing method, multiple communication messages are read from a distributed message queue storing communication messages based on message sorting parameters that reflect message sorting speed. These messages are then distributed to local queues corresponding to each third-party system. A second communication message is read from each local queue based on data transmission parameters that reflect data transmission speed, and sent to the corresponding third-party system for processing. The data transmission parameters for each local queue are determined based on the processing capacity parameters of the corresponding third-party system, ensuring a match between the two systems. Therefore, this method avoids system crashes caused by data transmission speeds exceeding the processing capacity of some third-party systems, thus improving system stability and overall data processing efficiency. Furthermore, the data transmission speed can be reasonably set according to the processing capacity of the third-party system to fully utilize system resources and improve resource utilization. In addition, two-layer flow control using message sorting parameters and data transmission parameters can achieve peak shaving and valley filling of traffic to third-party systems, improving overall system stability and resource utilization efficiency.

[0079] In one embodiment, the processing capacity parameters of the third-party system include the transaction volume per unit time, and the message sorting parameters include the first time window and the target number of reads within the first time window. For example... Figure 3 As shown, the method may also include a message sorting parameter determination process, specifically including the following steps:

[0080] Step 301: Obtain the transaction volume of each third-party system within a unit of time, and calculate the sum of the transaction volume within each unit of time.

[0081] In implementation, the transaction volume per unit time of the third-party system (e.g., transactions per second (tps)) can be stored as an attribute parameter of the third-party system. The distributed system can call relevant interfaces to send requests to the connected third-party systems to obtain the transaction volume per unit time of the third-party systems and calculate the sum of the transaction volumes per unit time (denoted as P). A transaction can be considered as the third-party system receiving a communication message from the distributed system, processing it, and then sending back the corresponding information.

[0082] Step 302: Based on the sum, the number of processing nodes in the distributed system, and the preset first time window, determine the target number of reads within the first time window.

[0083] In implementation, each distributed node in a distributed system can contain multiple processing nodes, and a processing node can be a single device (physical device or virtual device). For example... Figure 4 The distributed system shown contains N distributed nodes (set1 to setN), each of which can contain I devices (set1 contains 6 devices in this example), and each device is a processing node. Therefore, the distributed system will contain Q processing nodes, where Q = I * N. The first time window (denoted as T1) can be manually set as needed or automatically set according to other rules, such as 200ms. The distributed system can determine the target read quantity M1 of each processing node within the first time window T1 based on the sum P of the transaction processing volume of each third-party system within a unit of time, the number Q of processing nodes in the distributed system, and the preset first time window T1, such that the total number of communication messages sorted by the distributed system within a unit of time is greater than or equal to the sum of the transaction processing volume of the third-party systems within a unit of time. For example, if the first time window T1 is in milliseconds and the unit of time is in seconds, then the determined target read quantity M1 can satisfy the following formula:

[0084] P≤M1×Q×(1000 / T1)

[0085] Understandably, each processing node can be configured with multiple consumer threads, and each consumer thread can read the first communication message from the target distributed message queue in parallel. Therefore, based on the target read count M1 and the number of consumer threads C, the read count of each consumer thread can be determined, ensuring that the total number of communication messages read by all consumer threads within a processing node in the first time window T1 is consistent with the target read count M1.

[0086] Optionally, the target read count M1 can be set such that the total number of communication messages sorted by the distributed system per unit time is greater than 5% to 20% of the sum of the transaction processing volume of each third-party system per unit time.

[0087] In this embodiment, the message sorting parameters of the distributed system can be determined based on the total processing capacity of each third-party system, so that the number of messages stored in the local queue can match the processing capacity of each third-party system. In other words, the traffic can be reasonably controlled through the message sorting parameters to improve system stability and overall data processing efficiency. In addition, the capacity of the local queue can be reasonably set to improve memory resource utilization.

[0088] In one embodiment, the message sorting process in step 201 specifically includes the following steps: when the current time reaches the start time of the current first time window, read the target number of first communication messages from the target distributed message queue; determine the third-party system corresponding to each first communication message based on the feature information contained in each first communication message, and allocate each first communication message to the local queue corresponding to each third-party system based on the third-party system corresponding to each first communication message.

[0089] In implementation, each processing node in the distributed system can sort messages based on message sorting parameters (i.e., the first time window T1 and the target number of reads M1 within the first time window). Each processing node can process messages in parallel. Specifically, when the current time reaches the start time of the current first time window, each processing node can read the target number of communication messages from the target distributed message queue, determine the corresponding third-party system based on the characteristic information of each communication message, and then allocate the read communication messages to the local queue corresponding to the third-party system. The end time of the current first time window can be used as the start time of the next time window.

[0090] In this embodiment, each processing node of the distributed system can periodically execute message sorting steps based on a first time window and a target read quantity. Since the first time window (period) and the number of messages sorted within that time window are matched with the processing capabilities of each third-party system, on the one hand, system stability can be ensured through sorting flow control; on the other hand, the number of messages stored in the local queue and the number of messages consumed from the local queue can be kept in dynamic balance. Thus, the capacity of the local queue can be reasonably set so that it can meet the expected number of communication messages that match the processing capabilities of the third-party system while improving memory utilization.

[0091] In one embodiment, the processing capacity parameter of the third-party system includes the transaction processing volume per unit time, and the data transmission parameter includes the target transmission quantity within the second time window and the second time window. For example... Figure 5 As shown, another data processing method is provided, which further includes the following steps:

[0092] Step 501: For each local queue, obtain the transaction processing volume of the corresponding third-party system within a unit of time.

[0093] In implementation, the distributed system can call relevant interfaces to obtain the transaction volume per unit time of each third-party system, such as the transaction volume per second (tps).

[0094] Step 502: Determine the transaction processing volume of the local queue within a unit of time based on the transaction processing volume of the third-party system and the number of processing nodes in the distributed system.

[0095] In implementation, a distributed system can calculate the transaction volume (denoted as Si) of the local queue corresponding to the third-party system within a unit of time, based on the transaction volume of the third-party system within a unit of time (e.g., Pi) and the number of processing nodes Q in the distributed system. The goal is to ensure that the sum of the transaction volumes of the local queues corresponding to the same third-party system across all processing nodes is greater than or equal to the transaction volume of the third-party system within a unit of time. For example, if all processing nodes use the same data transmission parameters, the formula for calculating the transaction volume Si of the local queue within a unit of time can be as follows:

[0096] Si = Pi / Q

[0097] Understandably, different data transmission parameters can be used for each processing node. For example, different weights can be set for each processing node, and the weighted sum of the transaction processing volume of the local queue corresponding to the same third-party system in each processing node within a unit of time can be greater than or equal to the transaction processing volume of the third-party system within a unit of time.

[0098] Step 503: Based on the transaction processing volume per unit time corresponding to the local queue and the preset second time window, determine the target transmission quantity within the second time window corresponding to the local queue.

[0099] In implementation, each processing node in the distributed system can determine the target transmission quantity M2 within the second time window corresponding to its local queue based on the transaction processing volume Si per unit time and the preset second time window T2. This ensures that the data transmission rate corresponding to transmitting the target transmission quantity M2 communication messages within the second time window T2 is consistent with the data transmission rate reflected by the transaction processing volume Si per unit time. For example, if the second time window T2 is in milliseconds and the unit time is in seconds, then the determined target transmission quantity M2 can satisfy the following formula:

[0100] Si = M² × (1000 / T²)

[0101] In this embodiment, each processing node in the distributed system can use the second time window of the local queue corresponding to the third-party system and the target number of transmissions within the time window as data transmission parameters for processing flow control. Each processing node can process in parallel, ensuring that the total number of communication messages transmitted by each processing node to the same third-party system within the second time window matches the processing capacity of that third-party system. This improves system stability and overall data processing efficiency. Furthermore, data transmission parameters can be rationally set based on the processing capacity of the third-party system, fully leveraging the system's concurrent performance advantages.

[0102] In one embodiment, the data transmission process in step 202 specifically includes the following steps: for each local queue, when the current time reaches the start time of the current second time window corresponding to the local queue, read the target number of second communication messages from the local queue and send the read second communication messages to the third-party system corresponding to the local queue.

[0103] In implementation, each processing node of the distributed system can be configured with a processing thread corresponding to a third-party system, such as... Figure 4 As shown, communication messages can be read from the corresponding local queues through the processing threads of each third-party system. When the current time reaches the start time of the current second time window corresponding to the local queue, the processing thread can read the target transmission quantity M² of second communication messages from the local queue and send the read second communication messages to the third-party system corresponding to the local queue. It is understandable that the processing node can use one or more threads to send the read second communication messages to the third-party system corresponding to the local queue.

[0104] In this embodiment, each processing node in the distributed system may contain processing threads corresponding to each third-party system. These threads periodically execute data transmission steps based on the second time window of the local queue corresponding to the third-party system and the target transmission quantity within the time window. This decouples the connection threads of each third-party system, allowing for the reasonable setting of the data transmission speed corresponding to each third-party system, ensuring system stability and performance utilization, and improving overall data processing efficiency.

[0105] In one embodiment, the data transmission parameters include a second time window and the target number of transmissions within the second time window. For example... Figure 6 As shown, the data transmission process in step 202 specifically includes the following steps:

[0106] Step 601: For each local queue, if the current time reaches the start time of the current second time window corresponding to the local queue, determine whether there is an abnormal message in the abnormal message queue corresponding to the local queue.

[0107] The exception message queue stores communication messages whose processing result is an exception. After a processing node sends a communication message to a third-party system, it can determine the processing result of each message based on the feedback from the third-party system. The processing result includes normal processing and processing exception. The processing node can store communication messages with processing exceptions in the exception message queue.

[0108] In practice, for each local queue, each processing node in the distributed system can determine whether there are any abnormal messages in the abnormal message queue corresponding to that local queue when the current time reaches the start time of the current second time window corresponding to that local queue.

[0109] Step 602: If there are abnormal messages in the abnormal message queue, read the abnormal messages from the abnormal message queue.

[0110] In practice, if there are abnormal messages in the abnormal message queue, the processing node can read all or part of the abnormal messages from the abnormal message queue.

[0111] Step 603: If the first number of read exception messages is not less than the target transmission number, send the read exception messages to the third-party system corresponding to the local queue.

[0112] In practice, if the number of abnormal messages read by the processing node is not less than (greater than or equal to) the target transmission quantity M2 within the second time window, the processing node can send the read abnormal messages to the corresponding third-party system.

[0113] Step 604: If the first quantity is less than the target transmission quantity, determine the second quantity based on the first quantity and the target transmission quantity, and read the second quantity of second communication messages from the local queue.

[0114] In implementation, if the number of abnormal messages read by the processing node is less than the target transmission quantity M2 within the second time window, the difference between the number of abnormal messages and the target transmission quantity M2 can be calculated, which is the second quantity. Then, the processing node can read the second quantity of communication messages from the local queue. That is, the sum of the number of communication messages read from the local queue and the number of abnormal messages read is consistent with the target transmission quantity within the second time window.

[0115] Step 605: Send the read exception message and second communication message to the third-party system corresponding to the local queue.

[0116] In practice, the processing node can send the communication messages and exception messages read from the local queue to the third-party system corresponding to the local queue to complete the data transmission task within the current time window.

[0117] In this embodiment, communication messages with historical processing results indicating processing errors are stored as exception messages in an exception message queue. Within the current time window, exception messages can be retrieved from the exception message queue and resent to a third-party system for processing. This enables message retry, avoids abnormal processing results caused by network problems or other reasons, and improves the data processing success rate after multiple retries.

[0118] In one embodiment, the method further includes the following steps: for each local queue, based on feedback information from the third-party system corresponding to the local queue, determine the processing result of each communication message, and store communication messages whose processing result is a processing exception as exception messages in the exception message queue corresponding to the local queue.

[0119] In implementation, after sending communication messages to a third-party system, the processing node can receive feedback information from the third-party system and determine the processing result based on this feedback. Then, the processing node can store communication messages with a processing error as exception messages in the corresponding exception message queue in its local queue, enabling message retry and improving the data processing success rate. Optionally, the number of retries can be set. If the same communication message still fails to process after multiple retries, the communication message can be stored in the database for business personnel to query and review.

[0120] In one embodiment, the method may further include an abnormal data exceeding alarm step, specifically including: determining the ratio of the number of abnormal communication messages to the total number of communication messages processed, and outputting a first alarm message if the ratio is greater than a preset threshold.

[0121] In implementation, after the distributed system sends communication messages to a third-party system, it can determine the processing result based on the feedback information from the third-party system. If, within the same period (the second time window), the ratio of the number of communication messages with processing errors corresponding to a certain third-party system to the total number of communication messages sent to that third-party system for processing within that period is greater than a preset threshold (e.g., greater than 20%, i.e., a success rate lower than 80%), the distributed system can output a first alarm message to remind business personnel to troubleshoot the fault in a timely manner. Understandably, the proportion of processing errors or the success rate can also be calculated based on the processing results over multiple periods.

[0122] In this embodiment, the processing results of communication messages can be monitored. If the proportion of abnormal processing of communication messages corresponding to a certain third-party system is high (low success rate), the third-party system may have crashed or encountered other problems. Therefore, alarm information can be output to remind business personnel to intervene and troubleshoot the fault in a timely manner.

[0123] In one embodiment, the method may further include a queue length warning step, specifically including: for each local queue, if the number of communication messages in the local queue exceeds a preset threshold, outputting a second alarm message.

[0124] In implementation, the distributed system can monitor the number of messages stored in each local queue, i.e. the length of the queue. If the length is greater than the threshold (e.g., the length reaches 90% of the maximum capacity, or the length reaches 1000), a second alarm message will be output to remind business personnel to troubleshoot the fault.

[0125] In this embodiment, the length of the local queue can be monitored. If the length of the local queue exceeds the threshold, it means that the data transmission thread has not read the message from the local queue as expected and sent it to the third-party system. This may indicate that the local system or the third-party system has malfunctioned, and thus the business personnel can be promptly reminded to troubleshoot the fault.

[0126] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0127] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0128] In one embodiment, such as Figure 7 As shown, a data processing apparatus 700 is provided, including: an allocation module 701 and a transmission module 702, wherein:

[0129] The allocation module 701 is used to respond to the data transmission instruction, read multiple first communication messages from the target distributed message queue according to the message sorting parameters, and allocate each read first communication message to the local queue corresponding to each third-party system.

[0130] The sending module 702 is used to read a second communication message from each local queue according to the data transmission parameters corresponding to the local queue, and send the read second communication message to the third-party system corresponding to the local queue; the data transmission parameters corresponding to the local queue are determined according to the processing capability parameters of the third-party system corresponding to the local queue.

[0131] In one embodiment, the processing capacity parameters of the third-party system include the transaction processing volume per unit time, and the message sorting parameters include a first time window and the target read quantity within the first time window. The device further includes a first acquisition module and a first determination module, wherein:

[0132] The first acquisition module is used to acquire the transaction volume of each third-party system within a unit of time and calculate the sum of the transaction volume within each unit of time.

[0133] The first determination module is used to determine the target number of reads within the first time window based on the sum, the number of processing nodes in the distributed system, and a preset first time window.

[0134] In one embodiment, the allocation module 701 is specifically configured to: when the current time reaches the start time of the current first time window, read a target number of first communication messages from the target distributed message queue; determine the third-party system corresponding to each first communication message based on the feature information contained in each first communication message, and allocate each first communication message to the local queue corresponding to each third-party system based on the third-party system corresponding to each first communication message.

[0135] In one embodiment, the processing capacity parameter of the third-party system includes the transaction processing volume per unit time, and the data transmission parameter includes the target transmission quantity within the second time window and the second time window. The device further includes a second acquisition module, a second determination module, and a third determination module, wherein:

[0136] The second acquisition module is used to acquire the transaction processing volume of the corresponding third-party system within a unit of time for each local queue.

[0137] The second determining module is used to determine the transaction volume of the local queue within a unit of time based on the transaction volume of the third-party system within a unit of time and the number of processing nodes of the distributed system.

[0138] The third determination module is used to determine the target transmission quantity within the second time window corresponding to the local queue based on the transaction processing volume per unit time and the preset second time window.

[0139] In one embodiment, the sending module 702 is specifically configured to: for each local queue, when the current time reaches the start time of the current second time window corresponding to the local queue, read the target number of second communication messages from the local queue, and send the read second communication messages to the third-party system corresponding to the local queue.

[0140] In one embodiment, the data transmission parameters include a second time window and a target number of transmissions within the second time window. The sending module 702 is specifically configured to: for each local queue, if the current time reaches the start time of the current second time window corresponding to the local queue, determine whether there are any abnormal messages in the abnormal message queue corresponding to the local queue; the abnormal message queue is used to store communication messages whose processing result is a processing exception; if there are abnormal messages in the abnormal message queue, read the abnormal messages from the abnormal message queue; if the first number of read abnormal messages is not less than the target number of transmissions, send the read abnormal messages to the third-party system corresponding to the local queue; if the first number is less than the target number of transmissions, determine a second number based on the first number and the target number of transmissions, and read the second number of second communication messages from the local queue; send the read abnormal messages and second communication messages to the third-party system corresponding to the local queue.

[0141] In one embodiment, the device further includes a storage module, which is used to determine the processing result of each communication message for each local queue based on feedback information from the third-party system corresponding to the local queue, and store communication messages whose processing result is a processing exception as exception messages in the exception message queue corresponding to the local queue.

[0142] In one embodiment, the device further includes a first alarm module, used to determine the ratio of the number of communication messages with abnormal processing results to the total number of communication messages processed, and to output a first alarm message if the ratio is greater than a preset threshold.

[0143] In one embodiment, the device further includes a second alarm module for outputting a second alarm message for each local queue when the number of communication messages in the local queue exceeds a preset threshold.

[0144] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0145] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data required for or generated by the aforementioned data processing method. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0146] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0147] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0148] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0149] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0150] The data processing methods, apparatus, computer equipment, storage media, and computer program products provided in this application relate to the field of big data technology and can be used in the field of financial technology or other fields. This application does not limit the application field.

[0151] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0152] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0153] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: In response to the data transmission command, according to the message sorting parameters, multiple first communication messages are read from the target distributed message queue, and each read first communication message is allocated to the local queue corresponding to each third-party system; For each local queue, a second communication message is read from the local queue according to the data transmission parameters corresponding to the local queue, and the read second communication message is sent to the third-party system corresponding to the local queue; the data transmission parameters corresponding to the local queue are determined according to the processing capability parameters of the third-party system corresponding to the local queue. The processing capacity parameters of the third-party system include the transaction processing volume per unit time, and the message sorting parameters include a first time window and the target read quantity within the first time window; the method further includes: Obtain the transaction volume of each third-party system within a unit of time, and calculate the sum of the transaction volumes within each unit of time. Based on the sum, the number of processing nodes in the distributed system, and the preset first time window, the target number of reads within the first time window is determined. The data transmission parameters include a second time window and the target number of transmissions within the second time window; the method further includes: For each local queue, obtain the transaction volume of the corresponding third-party system within a unit of time. The transaction volume per unit time of the local queue is determined based on the transaction volume per unit time of the third-party system and the number of processing nodes of the distributed system. Based on the transaction volume per unit time corresponding to the local queue and the preset second time window, the target transmission quantity within the second time window corresponding to the local queue is determined.

2. The method according to claim 1, characterized in that, The step of reading multiple first communication messages from the target distributed message queue according to message sorting parameters, and distributing each read first communication message to the corresponding local queue of each third-party system, includes: If the current time reaches the start time of the current first time window, read the target number of first communication messages from the target distributed message queue; Based on the feature information contained in each of the first communication messages, the third-party system corresponding to each of the first communication messages is determined, and based on the third-party system corresponding to each of the first communication messages, each of the first communication messages is allocated to the local queue corresponding to each third-party system.

3. The method according to claim 1, characterized in that, The step of reading a second communication message from each local queue according to the data transmission parameters corresponding to the local queue, and sending the read second communication message to the third-party system corresponding to the local queue, includes: For each local queue, when the current time reaches the start time of the current second time window corresponding to the local queue, the target number of second communication messages are read from the local queue, and the read second communication messages are sent to the third-party system corresponding to the local queue.

4. The method according to any one of claims 1 to 3, characterized in that, The data transmission parameters include a second time window and the target number of transmissions within the second time window; the step of reading a second communication message from the local queue according to the data transmission parameters corresponding to the local queue, and sending the read second communication message to the third-party system corresponding to the local queue, includes: For each local queue, if the current time reaches the start time of the current second time window corresponding to the local queue, it is determined whether there is an abnormal message in the abnormal message queue corresponding to the local queue; the abnormal message queue is used to store communication messages whose processing result is a processing abnormality. If an exception message exists in the exception message queue, read the exception message from the exception message queue; If the first number of abnormal messages read is not less than the target transmission number, the read abnormal messages are sent to the third-party system corresponding to the local queue. If the first quantity is less than the target transmission quantity, a second quantity is determined based on the first quantity and the target transmission quantity, and the second quantity of second communication messages is read from the local queue; The retrieved exception message and second communication message are sent to the third-party system corresponding to the local queue.

5. The method according to claim 3, characterized in that, After sending the read second communication message to the third-party system corresponding to the local queue, the process further includes: For each local queue, the processing result of each communication message is determined based on the feedback information from the third-party system corresponding to the local queue, and communication messages whose processing result is an exception are stored as exception messages in the exception message queue corresponding to the local queue.

6. The method according to claim 4, characterized in that, After determining the processing result of each communication message based on the feedback information from the third-party system corresponding to the local queue, the process further includes: The processing result is determined as the ratio of the number of abnormal communication messages to the total number of communication messages processed, and if the ratio is greater than a preset threshold, a first alarm message is output.

7. The method according to claim 1, characterized in that, The method further includes: For each of the local queues, if the number of communication messages in the local queue exceeds a preset threshold, a second alarm message is output.

8. A data processing apparatus, characterized in that, The device includes: The allocation module is used to respond to data transmission instructions, read multiple first communication messages from the target distributed message queue according to message sorting parameters, and allocate each read first communication message to the local queue corresponding to each third-party system. The sending module is configured to, for each local queue, read a second communication message from the local queue according to the data transmission parameters corresponding to the local queue, and send the read second communication message to the third-party system corresponding to the local queue; the data transmission parameters corresponding to the local queue are determined according to the processing capability parameters of the third-party system corresponding to the local queue. The processing capacity parameters of the third-party system include the transaction processing volume per unit time, and the message sorting parameters include a first time window and the target read quantity within the first time window; the device further includes: The first acquisition module is used to acquire the transaction volume of each third-party system within a unit of time and calculate the sum of the transaction volume of each unit of time. The first determining module is used to determine the target number of reads within the first time window based on the sum, the number of processing nodes in the distributed system, and a preset first time window; The data transmission parameters include a second time window and the target number of transmissions within the second time window; the device further includes: The second acquisition module is used to acquire the transaction processing volume of the third-party system corresponding to each local queue within a unit of time for each local queue. The second determining module is used to determine the transaction volume of the local queue within a unit of time based on the transaction volume of the third-party system within a unit of time and the number of processing nodes of the distributed system. The third determining module is used to determine the target transmission quantity within the second time window corresponding to the local queue based on the transaction processing volume per unit time corresponding to the local queue and the preset second time window.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images