Intelligent data processing method and device and electronic equipment

Abstract

The invention relates to the technical field of computing, in particular to an intelligent data processing method and device and electronic equipment, and is used for solving the problem that a distributed system (such as a distributed bank system) cannot inherit an asynchronous processing function of a host bank core system. The method is applied to a distributed system and comprises the following steps: dividing data in a message sending schedule into N groups; for each group in the N groups, executing the following scanning steps: loading a distributed lock on the current group; if the distributed locking succeeds, M pieces of data included in the current group are read from the message sending schedule; and according to the progress data of the M pieces of data, reading the plurality of unprocessed asynchronous messages from the asynchronous message sending table, and sending the plurality of unprocessed asynchronous messages.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and electronic equipment for intelligent data processing. Background Art

[0002] As the core banking system gradually moves from the mainframe to the distributed system, the asynchronous processing functions originally performed on the mainframe (i.e. the mainframe banking core system) need to be taken over on the distributed system (i.e. the distributed system, such as the distributed banking system).

[0003] However, due to the different architectures of the distributed banking system and the host banking core system, the asynchronous processing function in the host banking core system is not applicable to the distributed banking system, making it impossible for the distributed banking system to inherit the asynchronous processing function of the host banking core system. Summary of the invention

[0004] The present application provides a data intelligent processing method, device and electronic device to solve the problem that a distributed system (such as a distributed banking system) cannot inherit the asynchronous processing function of the host bank core system. The specific implementation scheme is as follows:

[0005] In a first aspect, the present application provides a method for intelligent data processing, which is applied to a distributed system, and the method comprises:

[0006] Divide the data in the message sending schedule table into N groups; wherein N is a positive integer, and the message sending schedule table includes the progress data of the asynchronous messages in the asynchronous sending message table;

[0007] For each of the N groups, the following scanning steps are performed:

[0008] Apply a distributed lock to the current group; wherein the current group includes M pieces of data, where M is a positive integer;

[0009] If the above distributed lock is successful, read the M pieces of data from the message sending schedule;

[0010] According to the progress data of each of the M pieces of data, the unprocessed multiple asynchronous messages are read from the asynchronous message sending table respectively, and the unprocessed multiple asynchronous messages are sent.

[0011] Through the above-mentioned application embodiment, after the distributed system divides the data in the message sending schedule into N groups, for each of the N groups, after the upper distributed lock is successfully obtained, a scanning step is performed to process multiple unprocessed asynchronous messages in the asynchronous message sending table according to the progress data corresponding to the M data in the current group read from the message sending schedule, so that the distributed system can process asynchronous messages, thereby enabling the distributed system to have asynchronous processing capabilities, and further enabling the distributed system to inherit the asynchronous processing capabilities of the host bank core system.

[0012] In a possible implementation, dividing the data in the message sending schedule into N groups includes:

[0013] Acquire a configuration; wherein the configuration includes the number of partitions processed by the asynchronous sending thread at one time; the asynchronous sending thread is a thread created by the asynchronous sending unit in the distributed system for processing the asynchronous message;

[0014] All data in the message sending schedule are read, and the data in the message sending schedule are grouped according to the number of partitions to obtain the N groups.

[0015] Through the above-mentioned application embodiment, the data in the message sending schedule is grouped according to the number of partitions processed at one time by the asynchronous sending thread in the configuration, so that the number of data items included in each group is the number of partitions processed at one time by the asynchronous sending thread, which is beneficial to improving the processing efficiency of the asynchronous processing function of the distributed system.

[0016] In a possible implementation manner, for each of the N groups, the following scanning steps are performed, including:

[0017] Randomly select a group from the N groups as a starting point, and splice a data scanning plan based on the starting point and the N groups;

[0018] For the N groups, starting from the starting point, based on the data scanning plan, the following scanning steps are performed in sequence.

[0019] Through the above-mentioned application embodiment, a group is randomly selected from N groups as the starting point to splice the data scanning plan, and then starting from the selected starting point, the scanning steps are executed sequentially based on the data scanning plan, making it more likely that the starting points of different asynchronous sending threads are inconsistent, avoiding the problem of inefficiency caused by each asynchronous sending thread starting to execute the scanning step from the same starting point, thereby improving the processing efficiency of the asynchronous processing function of the distributed system.

[0020] In a possible implementation manner, after the distributed lock is applied to the current group, the method further includes:

[0021] If the above distributed lock fails, the current group is skipped, and the scanning step is continued for the groups that have not been processed among the N groups.

[0022] Through the above application embodiment, after the distributed lock fails, the current group is skipped, and the scanning step is continued for the groups that have not been processed in the N groups, further improving the asynchronous processing function of the distributed system. In addition, after determining that the distributed lock fails, it can be determined that there is already a thread processing the data in the current group, so the current group is skipped, ensuring that only one thread processes the data in the current group, and further ensuring that each piece of data is processed by only one thread.

[0023] In a possible implementation manner, after reading the unprocessed multiple asynchronous messages from the asynchronous message sending table respectively according to the progress data of each of the M pieces of data and sending the unprocessed multiple asynchronous messages, the method further includes:

[0024] The processing progress of the current group is written back to the message sending progress table.

[0025] Through the above application embodiment, after processing the data of the current group, the processing progress of the current group is written back to the message sending schedule, so that the latest processing progress can be recorded in real time in the message sending schedule to avoid the distributed system from repeating the processed data later.

[0026] In a possible implementation, the method further includes:

[0027] Start the daemon thread;

[0028] Based on the daemon thread, the status of the asynchronous sending thread is periodically detected.

[0029] Through the above-mentioned application embodiment, the status of the asynchronous sending thread is periodically detected based on the started daemon thread, so as to promptly detect problems arising in the processing of sending asynchronous messages, thereby solving the problems in a timely manner to avoid the impact of the problems on the processing of asynchronous messages and make the asynchronous processing function of the distributed system more perfect.

[0030] In a second aspect, the present application further provides a data intelligent processing device, which is applied to a distributed system, and includes:

[0031] A grouping module, used to divide the data in the message sending schedule into N groups; wherein N is a positive integer, and the message sending schedule includes the progress data of the asynchronous messages in the asynchronous sending message table;

[0032] The processing module is configured to perform the following scanning steps for each of the N groups:

[0033] Apply a distributed lock to the current group; wherein the current group includes M pieces of data, where M is a positive integer;

[0034] If the above distributed lock is successful, read the M pieces of data from the message sending schedule;

[0035] According to the progress data of each of the M pieces of data, the unprocessed multiple asynchronous messages are read from the asynchronous message sending table respectively, and the unprocessed multiple asynchronous messages are sent.

[0036] In a possible implementation, the grouping module is specifically used to obtain a configuration; wherein the configuration includes the number of partitions processed by the asynchronous sending thread at one time; the asynchronous sending thread is a thread created by the asynchronous sending unit in the distributed system for processing the asynchronous message; read all the data in the message sending schedule, and group the data in the message sending schedule according to the number of partitions to obtain the N groups.

[0037] In a possible implementation, the processing module is specifically used to randomly select a group from the N groups as a starting point, and splice a data scanning plan based on the starting point and the N groups; for the N groups, starting from the starting point, based on the data scanning plan, the following scanning steps are executed in sequence.

[0038] In a possible implementation, the processing module is further configured to skip the current group if the above distributed lock fails, and continue to execute the scanning step for the groups that have not been processed among the N groups.

[0039] In a possible implementation, the processing module is further configured to write back the processing progress of the current group to the message sending progress table.

[0040] In a possible implementation, the device further includes a daemon module, and the daemon module is used to start a daemon thread; based on the daemon thread, the state of the asynchronous sending thread is periodically detected.

[0041] In a third aspect, the present application provides an electronic device, including:

[0042] Memory, used to store computer programs;

[0043] The processor is used to implement the above-mentioned steps of the intelligent data processing method when executing the computer program stored in the memory.

[0044] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for intelligent data processing are implemented.

[0045] In a fifth aspect, the present application provides a computer program product, comprising: a computer program code, when the computer program code is run on a computer, the computer executes any one of the methods described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of the deployment structure of a distributed system provided in an embodiment of the present application;

[0047] Figure 2 A flowchart of a method for intelligent data processing provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a processing process of an intelligent data processing method provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of a data intelligent device provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0052] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0053] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0054] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0055] Before introducing the intelligent data processing method provided in the embodiment of the present application, in order to facilitate understanding, the technical background of the embodiment of the present application is first introduced in detail below.

[0056] The asynchronous processing function of the current host bank core system can be handled through the following solutions when processing asynchronous messages:

[0057] During online transactions, messages that need to be sent asynchronously (i.e., asynchronous messages) are generated and registered in the message flow table. Then, the process in the host bank core system scans the asynchronous messages in the message flow table according to the scanning progress, and processes the data in the message flow table (i.e., asynchronous messages) in the order in which they are scanned.

[0058] However, due to the different architectures of distributed systems (such as distributed banking systems) and host bank core systems, distributed systems cannot implement asynchronous processing functions through the solutions used by the above-mentioned host bank core systems to implement asynchronous processing functions, and thus cannot inherit the asynchronous processing functions of the host bank core systems.

[0059] Therefore, the present application proposes an intelligent data processing method, which can be applied to distributed systems, divides the data in the message sending schedule into N groups, and then performs a scanning step for each group. The scanning step is: for the current group, a distributed lock is applied. If the distributed lock is applied successfully, M data in the current group are read from the message sending schedule, and then according to the progress data of each of the M data, multiple unprocessed asynchronous messages are read from the asynchronous message sending table respectively, and the asynchronous messages are sent, thereby realizing the processing of asynchronous messages, so that the distributed system can realize the asynchronous processing function in this way, so that the distributed system has the asynchronous processing function, and then inherits the asynchronous processing function of the host bank core system.

[0060] In the implementation of this application, the above-mentioned distributed system can be a distributed banking system. The distributed system (such as a distributed banking system) is a system with scalability, high availability and other characteristics, and is composed of a cluster of multiple small physical machines to provide services. The distributed system deploys a unit structure, which is as follows: Figure 1 shown.

[0061] From this Figure 1 It can be seen that the distributed system deploys a business processing cluster for processing online business, an asynchronous message sending unit and a database. The business processing cluster and the asynchronous message sending unit use the same database and access the same data source.

[0062] Before the transaction of the online transaction ends (i.e., at the end of the transaction of the online transaction), the business processing cluster inserts the data to be sent (i.e., the message to be sent asynchronously) into the asynchronous message sending table, which is located in the database. In other words, at the end of the transaction of the online transaction, the distributed system inserts the message to be sent asynchronously into the asynchronous message sending table through the business processing cluster, i.e., inserts the asynchronous message, so that the distributed system can process the message to be sent asynchronously.

[0063] The data in the asynchronous message sending table can be subsequently processed by the asynchronous message sending unit by reading data. In other words, the distributed system processes the data in the asynchronous message sending table by reading data through the asynchronous message sending unit, thereby realizing the processing of asynchronous messages.

[0064] Furthermore, based on the business processing cluster, the asynchronous message sending unit and the database, the distributed system is equipped with the function of processing asynchronous messages.

[0065] The above asynchronous message sending table may include multiple data, and each data includes a legal person identity identification code (English: Identity document, abbreviated as ID), brand number, processing category, transaction date, transaction time, partition number, global event tracking number, message tracking number, message data, but is not limited to this. The basic structure of the asynchronous message sending table can be shown in Table 1 below.

[0066] Table 1

[0067]

[0068] The meanings of the legal person ID, brand, processing category, transaction date, transaction time, partition number, global event tracking number, message tracking number, and message data in the asynchronous message sending table are as described in Table 1 above, and will not be repeated here. Thus, according to the legal person ID, brand, processing category, transaction date, transaction time, partition number, global event tracking number, message tracking number, and message data in the asynchronous message sending table, various information of the message that needs to be sent asynchronously (i.e., asynchronous message) can be obtained.

[0069] In addition, by sending the partition number in the asynchronous message table, the business data can be effectively distributed as evenly as possible to each shard of the data.

[0070] In the embodiment of the present application, the database deployed in the distributed system may include a message sending progress table in addition to the above-mentioned asynchronous message sending table. The message sending progress table may include progress data of the asynchronous messages in the asynchronous message sending table.

[0071] The message sending schedule table may include multiple data, and each data includes legal person ID, brand, processing category, partition number, scan partition number initial value, scan partition number end value, scan timestamp, message sending table legal person ID, message sending table brand, message sending table processing category, message sending table partition number, message sending table transaction timestamp. And each data in the message sending schedule table corresponds to a database partition in the asynchronous message sending table, such as the range included in the scan partition number initial value and the scan partition number end value in the message sending schedule table corresponds to a database partition in the asynchronous message sending table. Then the basic structure of the message sending schedule table may be shown in Table 2 below.

[0072] Table 2

[0073]

[0074]

[0075] The meanings of the legal person ID, brand number, processing category, partition number, scanning partition number initial value, scanning partition number end value, scanning timestamp, message sending table legal person ID, message sending table brand number, message sending table processing category, message sending table partition number, and message sending table transaction timestamp in the message sending schedule table are shown in the above Table 2, which will not be repeated here. Therefore, the information of each piece of data in the message sending schedule table can be obtained according to the legal person ID, brand number, processing category, partition number, scanning partition number initial value, scanning partition number end value, scanning timestamp, message sending table legal person ID, message sending table brand number, message sending table processing category, message sending table partition number, and message sending table transaction timestamp, so as to facilitate subsequent processing based on the information.

[0076] In the basic structure of the message sending schedule shown in Table 2 above, the transaction timestamp of the message sending schedule can be based on the transaction date and transaction time in the corresponding message sending schedule.

[0077] Taking the examples of Table 1 and Table 2 above as an example, 20240801 in the transaction timestamp of the message sending table in Table 2 is the transaction date in Table 1, and 125200000 in the timestamp of the message sending table in Table 2 is the transaction time in Table 1.

[0078] In the basic structure of the message sending schedule shown in Table 2 above, the message sending table legal person ID, message sending table brand number, message sending table processing category, message sending table partition number, and message sending table transaction timestamp represent the progress data, so that the processing progress of the data in the message sending schedule can be clarified based on the progress data.

[0079] In the basic structure of the message sending schedule shown in Table 2 above, the legal person ID, brand number, processing category, partition number, scan partition number initial value, and scan partition number end value can also be located in a part of the partition primary key value.

[0080] The specific process of the solution for realizing asynchronous processing function by the distributed system (i.e., a data intelligent processing method provided by the present application) can be as follows: Figure 2 See Figure 2 The flowchart of a method for intelligent data processing provided by an embodiment of the present application is shown, and the method includes:

[0081] S201, dividing the data in the message sending schedule into N groups.

[0082] Wherein, N is a positive integer.

[0083] In the embodiment of the present application, the data in the message sending schedule is divided into N groups, which can be divided in the following manner:

[0084] First, the configuration is obtained, which may include the number of partitions and threads (such as the number of asynchronous sending threads) processed at one time by the asynchronous sending thread, but is not limited to this. Then all the data in the message sending schedule are read, and the data in the message sending schedule is grouped according to the number of partitions to obtain N groups, so that the number of data items contained in each of the N groups is the number of partitions processed at one time by the asynchronous sending thread, which is conducive to improving the processing efficiency of the asynchronous processing function of the distributed system.

[0085] Exemplarily, the message sending schedule includes 100 data items, and the number of partitions processed at one time in the configuration is 10 items. Then, the data in the message sending schedule can be divided into 10 groups, that is, N is 10.

[0086] S202: Apply a distributed lock to the current group among the N groups.

[0087] In the embodiment of the present application, the distributed lock may be a remote dictionary service (English: Remote Dictionary Server, abbreviated as Redis) distributed lock, but is not limited thereto, and a specific distributed lock may be selected according to a specific application scenario.

[0088] In order to improve the processing efficiency of each of the N groups, the distributed lock on the current group in the N groups can be applied by first randomly selecting a group from the N groups as the starting point, and then splicing the data scanning plan based on the starting point and the N groups. Then, for the N groups, starting from the starting point, the scanning steps are performed in sequence based on the data scanning plan. The scanning step includes steps S202-S204. Then, according to the data scanning plan and the starting point, the current group is determined, and a distributed lock is applied to the current group.

[0089] For example, N is 101 and the random starting point is 58, then distributed locks are applied to the N groups in the order of 59th to 100th and 0th to 58th.

[0090] Therefore, by executing the scanning step based on a random starting point, the amount of data processed by each asynchronous sending thread at the same time is much greater than the amount of data processed when each asynchronous sending thread starts executing the scanning step from the same starting point, thereby greatly improving the processing efficiency of the asynchronous processing function of the distributed system.

[0091] If the distributed lock is successful, it means that no other thread is processing the data of all partitions in the current group, and step S203 is executed.

[0092] If the distributed lock fails, it means that other threads are processing the data of all partitions in the current group, and step S204 is executed at this time.

[0093] S203, reading M pieces of data in the current group from the message sending progress table, reading multiple unprocessed asynchronous messages from the asynchronous message sending table according to the progress data of each of the M pieces of data, and sending the multiple unprocessed asynchronous messages.

[0094] After confirming that the distributed lock is successful, the unprocessed data in the asynchronous message sending table can be scanned and processed according to the latest processed data in the message sending schedule table.

[0095] Specifically, first, M pieces of data in the current group are read from the message sending progress table. Each piece of data in the M pieces of data includes legal person ID, brand number, processing category, partition number, scan partition number initial value, scan partition number end value, scan timestamp, message sending table legal person ID, message sending table brand number, message sending table processing category, message sending table partition number, and message sending table transaction timestamp.

[0096] Then, based on the progress data of each of the M data, i.e., the legal person ID of the message sending table, the message sending table brand number, the message sending table processing category, the message sending table partition number, and the message sending table transaction timestamp corresponding to each of the M data, the processing progress of each of the M data, i.e., the latest processed data in each data, can be determined, so that for each of the M data, the latest processed data is used as the starting message point of its message processing, and then for each of the M data, multiple asynchronous messages after the starting message point are read from the asynchronous message sending table, and the multiple asynchronous messages are sent. The multiple asynchronous messages are asynchronous messages that have not been processed.

[0097] Exemplarily, taking one piece of data among M pieces of data as an example, the piece of data is shown in Example 1 in Table 3 below.

[0098] Table 3

[0099] Field Name Example 1 Legal person ID CN000 Brand 01 Processing categories 00001 Partition Number 001 Scan partition number initial value 0000000 Scan partition number end value 1000000 Scan timestamp 20240801125300000 Message sending table legal person ID CN000 Message sending plate number 01 Message sending table processing category 00001 Message sending table partition number 0000300 Message sending table transaction timestamp 20240801125200000

[0100] Since the partition number of the message sending table indicates the partition number of the most recently processed message under the partition of the message sending table, it can be known from the partition number of the message sending table (0000300) that the data between 0000000 and 0000300 has been processed in the data range from the initial value of the scan partition number (0000000) to the end value of the scan partition number (1000000), and the partition number of the message sending table (0000300) is used as the starting message point for processing the message. Then, multiple asynchronous messages after the starting message point are read from the asynchronous message sending table, that is, multiple asynchronous messages after the partition number (0000300) of the message sending table are read from the asynchronous message sending table. However, it should be noted that the partition numbers corresponding to the multiple asynchronous messages read should be between the initial value of the scan partition number and the end value of the scan partition number, and should be the messages in the asynchronous message sending table corresponding to Example 1.

[0101] Optionally, the number of the multiple asynchronous messages read from the asynchronous message sending table may be determined according to a specific application scenario.

[0102] Optionally, when sending multiple asynchronous messages, each time an asynchronous message is sent, the processing result of the asynchronous message is written back to the asynchronous message sending table, so that the processing result of the asynchronous message is recorded in the asynchronous message sending table in time, so that the distributed system can be informed of the processing status of the asynchronous message in time.

[0103] Optionally, after all M pieces of data have been processed, that is, according to the progress data of each of the M pieces of data, multiple unprocessed asynchronous messages are read from the asynchronous message sending table respectively, and after sending the multiple unprocessed asynchronous messages, the processing progress of the current group is written back to the message sending schedule table to facilitate updating the progress data of the data in the message sending schedule table, so that when the group of data is processed subsequently, the distributed system can know where to start processing the data, thereby avoiding repeated processing of processed data.

[0104] Furthermore, after all M pieces of data have been processed, that is, according to the progress data of each of the M pieces of data, the unprocessed multiple asynchronous messages are read from the asynchronous message sending table respectively, and after the unprocessed multiple asynchronous messages are sent, according to the data scanning plan, the group that has not been processed among the N groups is re-determined as the current group, and then the scanning step is performed for the re-determined current group.

[0105] Specifically, it is determined whether there is any group among the N groups that has not been processed.

[0106] If there are groups that have not been processed, the data scanning plan is continued, the groups that have not been processed are taken as the current groups, and the scanning steps are performed for the re-determined current groups until all N groups are processed.

[0107] If there is no group that has not been processed, the data scanning plan is re-planned, and then the above steps S201-S204 are executed according to the re-planned new data scanning plan, so as to continue to process the asynchronous messages that have not been processed in the asynchronous message sending table.

[0108] S204, skip the current group, and take the group that has not been processed among the N groups as the current group.

[0109] In an embodiment of the present application, when the upper distributed lock fails and the current group is skipped, the group that has not been processed among the N groups is continued to be used as the current group, and the specific process of executing the scanning step for the re-determined current group is consistent with the specific process of continuing to execute the scanning step for the group that has not been processed among the N groups after processing the M data in the current group in the above step S203, and will not be repeated here.

[0110] In addition, in an embodiment of the present application, the distributed system can also create multiple asynchronous sending threads according to the configuration of different processing categories through the asynchronous processing unit, and then start multiple asynchronous sending threads, so as to execute the above steps S201-S205 based on multiple asynchronous sending threads, so that multiple asynchronous sending threads can execute the above steps S201-S205 at the same time, thereby saving the processing time of unprocessed asynchronous messages in the asynchronous message sending table, and further improving the efficiency of the asynchronous processing function of the distributed system.

[0111] Optionally, after starting all asynchronous sending threads, each machine in the distributed system can also start a daemon thread, and then based on the daemon thread, periodically detect the status of the asynchronous sending threads, so as to promptly discover problems that arise in the processing of asynchronous messages and solve the problems in a timely manner to avoid the impact of the problems on the processing of asynchronous messages and make the asynchronous processing function of the distributed system more complete.

[0112] Exemplarily, if the daemon thread detects the existence of a terminated asynchronous sending thread, such as an asynchronous sending thread A reporting an error, a new asynchronous sending thread is started to continue executing the task that the terminated asynchronous sending thread failed to complete.

[0113] In an embodiment of the present application, each machine in the distributed system may correspond to a daemon thread, so as to detect the status of the asynchronous sending thread in each machine respectively.

[0114] In summary, the intelligent data processing method proposed in the present application is that the distributed system (such as a distributed banking system) first divides the data in the message sending schedule into N groups, and then performs the following scanning steps for each group in turn. Specifically, for the current group, a distributed lock is applied. If the distributed lock is successful, M data in the current group are read from the message sending schedule, and then according to the progress data of each of the M data, multiple unprocessed asynchronous messages are read from the asynchronous message sending table respectively, and the asynchronous messages are sent, thereby realizing the processing of asynchronous messages.

[0115] Based on the unitized deployment structure of the distributed system, the distributed system inherits the asynchronous processing function (i.e. the function of asynchronously sending messages) of the core system of the host bank. It also supports the scalability and high availability of the distributed bank while ensuring that the functions remain unchanged.

[0116] The technical solution of this application is further explained below in conjunction with a specific application process.

[0117] like Figure 3The figure shows a processing process diagram of the intelligent data processing method. Taking the distributed banking system in the distributed system as an example, first, in the pre-processing module, the distributed banking system inserts the data to be sent (i.e., asynchronous message) into the asynchronous message sending table in the database through the business processing cluster at the end of the online transaction. Then, the distributed banking system records the processing progress of the asynchronous message in the asynchronous message sending table in the message sending progress table in the database.

[0118] In the planning module, the distributed banking system reads all the data in the message sending schedule, and groups the data in the message sending schedule according to the number of partitions processed at one time by the asynchronous sending thread in the configuration to obtain N groups; then randomly selects a group from the N groups as the starting point, and splices the starting point and the N groups into a data scanning plan. Then, the data scanning plan is transmitted to the scanning module.

[0119] In the scanning module, based on the data scanning plan transmitted from the planning module, the following scanning steps are performed in sequence.

[0120] Specifically, a Redis distributed lock is set for the current group in the data scanning plan.

[0121] If the Redis distributed lock is successful, the message is processed. Specifically, the data corresponding to the current group (i.e., the M data in the current group) is read from the message sending progress table, and then according to the progress data of each of the M data, multiple unprocessed asynchronous messages are read from the asynchronous message sending table respectively, and the unprocessed asynchronous messages are sent one by one according to the read data, and the processing results are written back to the asynchronous message sending table until the M data are processed, and the processing progress is written back to the message sending progress table, and then it is determined whether there is an unprocessed group among the N groups.

[0122] If the Redis distributed lock fails, the current group is skipped and it is determined whether there is any unprocessed group among the N groups.

[0123] If there are still unprocessed groups among the N groups, the scanning step is continued to be performed for the unprocessed groups among the N groups, that is, the new group is taken as the current group, and the above scanning step is continued to be performed for the current group.

[0124] If each of the N groups has been processed, that is, there is no unprocessed group in the N groups, then return to the planning module, randomly select the starting point again, plan the data scanning plan, and then continue to execute the scanning step based on the scanning module according to the re-planned new data scanning plan.

[0125] Based on the same inventive concept, the present application also provides an intelligent data processing device, such as Figure 4The figure shows a schematic diagram of the structure of a data intelligent processing device provided by the present application, which can be applied to a distributed system, and includes:

[0126] The grouping module 401 is used to divide the data in the message sending schedule into N groups; wherein N is a positive integer, and the message sending schedule includes the progress data of the asynchronous messages in the asynchronous sending message table;

[0127] The processing module 402 is configured to perform the following scanning steps for each of the N groups:

[0128] Apply a distributed lock to the current group; wherein the current group includes M pieces of data, where M is a positive integer;

[0129] If the above distributed lock is successful, read the M pieces of data from the message sending schedule;

[0130] According to the progress data of each of the M pieces of data, the unprocessed multiple asynchronous messages are read from the asynchronous message sending table respectively, and the unprocessed multiple asynchronous messages are sent.

[0131] In one possible implementation, the grouping module 401 is specifically used to obtain a configuration; wherein the configuration includes the number of partitions processed by the asynchronous sending thread at one time; the asynchronous sending thread is a thread created by the asynchronous sending unit in the distributed system to process the asynchronous message; all data in the message sending schedule is read, and the data in the message sending schedule is grouped according to the number of partitions to obtain the N groups.

[0132] In one possible implementation, the processing module 402 is specifically used to randomly select a group from the N groups as a starting point, and splice a data scanning plan based on the starting point and the N groups; for the N groups, starting from the starting point, based on the data scanning plan, the following scanning steps are executed in sequence.

[0133] In a possible implementation, the processing module 402 is further configured to skip the current group if the above distributed lock fails, and continue to execute the scanning step for the groups that have not been processed among the N groups.

[0134] In a possible implementation, the processing module 402 is further configured to write back the processing progress of the current group to the message sending progress table.

[0135] In a possible implementation, the device further includes a daemon module, and the daemon module is used to start a daemon thread; based on the daemon thread, the state of the asynchronous sending thread is periodically detected.

[0136] Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application. The electronic device can realize the function of the intelligent processing device of the aforementioned data. Figure 5 , the electronic equipment includes:

[0137] At least one processor 501, and a memory 502 connected to the at least one processor 501. The specific connection medium between the processor 501 and the memory 502 is not limited in the embodiment of the present application. Figure 5 In the example, the processor 501 and the memory 502 are connected via a bus 500. The bus 500 is Figure 5 The connections between other components are shown in bold lines, and are not intended to be limiting. The bus 500 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 501 can also be called a controller, and there is no limitation on the name.

[0138] In the embodiment of the present application, the memory 502 stores instructions that can be executed by at least one processor 501. The at least one processor 501 can execute the intelligent data processing method discussed above by executing the instructions stored in the memory 502. The processor 501 can implement Figure 4 The functions of each module in the device shown.

[0139] Among them, the processor 501 is the control center of the device, and can use various interfaces and lines to connect the various parts of the entire control device. By running or executing instructions stored in the memory 502 and calling data stored in the memory 502, the various functions of the device and processing data, the device can be monitored as a whole.

[0140] In one possible design, the processor 501 may include one or more processing units, and the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 501. In some embodiments, the processor 501 and the memory 502 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.

[0141] Processor 501 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the intelligent processing method of data disclosed in the embodiments of the present application can be directly embodied as a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0142] The memory 502 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 502 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (English: Random Access Memory, abbreviated as RAM), a static random access memory (English: Static Random Access Memory, abbreviated as SRAM), a programmable read-only memory (English: Programmable Read Only Memory, abbreviated as PROM), a read-only memory (English: Read Only Memory, abbreviated as ROM), an electrically erasable programmable read-only memory (English: Electrically Erasable Programmable Read-Only Memory, abbreviated as EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 502 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 502 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0143] By designing and programming the processor 401, the code corresponding to the intelligent data processing method described in the above embodiment can be fixed into the chip, so that the chip can execute the intelligent data processing method when running. Figure 2 The steps of the data intelligent processing method of the embodiment shown are as follows: How to design and program the processor 501 is a technology well known to those skilled in the art and will not be described in detail here.

[0144] Based on the same inventive concept, an embodiment of the present application further provides a storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the intelligent data processing method discussed above.

[0145] In some possible implementations, various aspects of the intelligent data processing method provided in the present application can also be implemented in the form of a program product, which includes program code. When the program product is run on an apparatus, the program code is used to enable the control device to execute the steps of the intelligent data processing method according to various exemplary embodiments of the present application described above in this specification.

[0146] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0147] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0148] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0150] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data intelligent processing method, applied to a distributed system, characterized in that: include: Divide the data in the message sending schedule table into N groups; wherein N is a positive integer, and the message sending schedule table includes the progress data of the asynchronous messages in the asynchronous sending message table; For each of the N groups, the following scanning steps are performed: Apply a distributed lock to the current group; wherein the current group includes M pieces of data, where M is a positive integer; If the above distributed lock is successful, read the M pieces of data from the message sending schedule; According to the progress data of each of the M pieces of data, the unprocessed multiple asynchronous messages are read from the asynchronous message sending table respectively, and the unprocessed multiple asynchronous messages are sent.

2. The method according to claim 1, characterized in that The data in the message sending schedule is divided into N groups, including: Acquire a configuration; wherein the configuration includes the number of partitions processed by the asynchronous sending thread at one time; the asynchronous sending thread is a thread created by the asynchronous sending unit in the distributed system for processing the asynchronous message; All data in the message sending schedule are read, and the data in the message sending schedule are grouped according to the number of partitions to obtain the N groups.

3. The method according to claim 1, characterized in that For each of the N groups, the following scanning steps are performed, including: Randomly select a group from the N groups as a starting point, and splice a data scanning plan based on the starting point and the N groups; For the N groups, starting from the starting point, based on the data scanning plan, the following scanning steps are performed in sequence.

4. The method according to claim 1, characterized in that After the distributed lock is applied to the current group, the following steps are further included: If the above distributed lock fails, the current group is skipped, and the scanning step is continued for the groups that have not been processed among the N groups.

5. The method according to claim 1, characterized in that After reading the unprocessed multiple asynchronous messages from the asynchronous message sending table according to the progress data of each of the M pieces of data, and sending the unprocessed multiple asynchronous messages, the method further includes: The processing progress of the current group is written back to the message sending progress table.

6. The method according to claim 1, characterized in that The method further comprises: Start the daemon thread; Based on the daemon thread, the status of the asynchronous sending thread is periodically detected.

7. An intelligent data processing device, applied to a distributed system, characterized in that: include: A grouping module, used to divide the data in the message sending schedule into N groups; wherein N is a positive integer, and the message sending schedule includes the progress data of the asynchronous messages in the asynchronous sending message table; The processing module is configured to perform the following scanning steps for each of the N groups: Apply a distributed lock to the current group; wherein the current group includes M pieces of data, where M is a positive integer; If the above distributed lock is successful, read the M pieces of data from the message sending schedule; According to the progress data of each of the M pieces of data, the unprocessed multiple asynchronous messages are read from the asynchronous message sending table respectively, and the unprocessed multiple asynchronous messages are sent.

8. The device according to claim 7, characterized in that The grouping module is further used to obtain a configuration; wherein the configuration includes the number of partitions processed by the asynchronous sending thread at one time; the asynchronous sending thread is a thread created by the asynchronous sending unit in the distributed system and used to process the asynchronous message; All data in the message sending schedule are read, and the data in the message sending schedule are grouped according to the number of partitions to obtain the N groups.

9. The device according to claim 7, characterized in that The processing module is further used to randomly select a group from the N groups as a starting point, and splice a data scanning plan based on the starting point and the N groups; For the N groups, starting from the starting point, based on the data scanning plan, the following scanning steps are performed in sequence.

10. The device according to claim 7, characterized in that The processing module is further configured to skip the current group and continue to execute the scanning step for the groups that have not been processed among the N groups if the above distributed lock fails.

11. The device according to claim 7, characterized in that The processing module is further configured to write back the processing progress of the current group to the message sending progress table.

12. The device according to claim 7, characterized in that The device also includes a daemon module, and the daemon module is used to start a daemon thread; Based on the daemon thread, the status of the asynchronous sending thread is periodically detected.

13. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to implement the method steps of any one of claims 1 to 6 when executing the computer program stored in the memory.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 6 are implemented.

15. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 6.

Images