Data processing method, device and storage medium

Abstract

The present invention discloses a data processing method, device and storage medium: a big data platform module incrementally extracts the business main table and associated business sub-tables of each business system, and pushes them to the billing database to form corresponding billing main tables and billing sub-tables; a monitoring module monitors the insert-type binary log corresponding to the billing main table in the billing database, and sends the monitored binary log to a transfer module via a message queue; the transfer module obtains the business main table ID based on the billing main table in the message queue, and searches the billing database for the billing sub-table associated with the billing main table based on the business main table ID; the data of the billing main table and the associated billing sub-table are combined into a message, and a message message queue containing the data message content is sent to the module; the message data is converted into a billing standard order based on the content of the message message queue. The present invention is used to complete data docking of multiple upstreams without affecting the upstream, and transfer business system documents into billing standard orders required by the billing system.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device and storage medium. Background Art

[0002] The logistics billing system needs to use a unified document (standard billing document) to calculate various types of logistics costs. However, logistics costs are generated at various nodes in the logistics process. To calculate these costs, it is necessary to connect to multiple different business systems. The data of the business systems has different forms. During the connection process, it is necessary to ensure that the performance of the business system is not consumed as much as possible and to adapt to the continuous changes in upstream business. The existing technology mainly relies on business personnel to extract business system data and manually import the data into the billing system. This requires constant verification and data errors cannot be avoided. A lot of manual intervention is required to ensure normal billing. Another way is for the billing system to provide an interface, allowing the business system to call and transmit data through the interface.

[0003] The existing front-end import method cannot cope with special business rules, cannot guarantee the accuracy of data, and manual labor cannot cope with the data pressure caused by the massive growth of business. Summary of the Invention

[0004] The purpose of the present invention is to provide a data processing method and device, in order to solve the problem of completing data docking of multiple upstreams without affecting the upstreams, and converting business system documents into billing standard documents required by the billing system.

[0005] To achieve the above-mentioned object of the invention, the present invention provides a service processing method, which includes:

[0006] The big data platform module incrementally extracts the main business table and related business sub-tables of each business system, and pushes them to the billing database to form the corresponding billing main table and billing sub-tables;

[0007] The monitoring module monitors the insert-type binary logs corresponding to the billing master table in the billing database and sends the monitored binary logs to the order transfer module through the message queue;

[0008] The order transfer module obtains the business main table ID based on the billing main table in the message queue, and searches the billing sub-table associated with the billing main table from the billing database based on the business main table ID; composes the data of the billing main table and the associated billing sub-table into a message, and sends a message message queue containing the data message content to itself; and converts the message data into a billing standard order based on the content of the message message queue.

[0009] To achieve the above-mentioned object, the present invention further provides a data processing device, which is applied to a billing system, comprising:

[0010] The big data platform module incrementally extracts the main business table and related business sub-tables of each business system and pushes them to the billing database to form the corresponding billing main table and billing sub-tables;

[0011] The monitoring module monitors the insert-type binary logs corresponding to the billing master table in the billing database and sends the monitored binary logs to the order transfer module through the message queue;

[0012] The order transfer module obtains the business main table ID based on the billing main table in the message queue, and searches the billing sub-table associated with the billing main table from the billing database based on the business main table ID; composes the data of the billing main table and the associated billing sub-table into a message, and sends a message message queue containing the data message content to itself; and converts the message data into a billing standard order based on the content of the message message queue.

[0013] In summary, the data processing method and device provided by the present invention are connected to multiple business systems. The big data platform module incrementally extracts the business main table and the associated business sub-table of each business system, and pushes them to the billing database to form the corresponding billing main table and billing sub-table; the monitoring module monitors the insert type binary log corresponding to the billing main table in the billing database, and sends the monitored binary log to the order transfer module through the message queue (MQ); the order transfer module obtains the business main table ID according to the billing main table in the MQ, and finds the billing sub-table associated with the billing main table from the billing database according to the business main table ID; the data of the billing main table and the associated billing sub-table are combined into a message, and a message MQ containing the data message content is sent to itself; the message data is converted into a billing standard order according to the content of the message MQ. Through the solution of the present invention, compared with the existing technology, the accuracy of the data is guaranteed, and it can cope with the data pressure generated by the large-scale growth of business. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Schematic diagram of the structure of the data processing device of the present invention.

[0015] Figure 2 Schematic diagram of the data processing method of the present invention.

[0016] Figure 3 The figure is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the solutions of the present invention are further described in detail below with reference to the accompanying drawings and embodiments.

[0018] The big data platform module in the present invention adopts the method of incremental extraction from the business system database. Incremental extraction only extracts the newly added or modified data in the table to be extracted in the database since the last extraction. Incremental extraction is more widely used than full extraction. How to capture the changed data is the key to incremental extraction. There are generally two requirements for the capture method: accuracy, which means being able to accurately capture the changed data in the business system at a certain frequency; performance, which means not putting too much pressure on the business system and affecting the existing business. At present, the commonly used methods for capturing changed data in incremental data extraction are: A. Triggers, which establish the required triggers on the table to be extracted. Generally, three triggers are established: insert, modify, and delete. Whenever the data in the source table changes, the corresponding trigger writes the changed data into a temporary table. The extraction thread extracts data from the temporary table, and the data extracted from the temporary table is marked or deleted. The advantage of the trigger method is that the data extraction performance is higher. The disadvantage is that it requires the establishment of triggers in the business table, which has a certain impact on the business system. B. Timestamp: This method captures change data based on snapshot comparisons. A timestamp field is added to the source table. When the system updates or modifies table data, the timestamp field's value is also updated. When extracting data, the system time is compared with the timestamp field's value to determine which data to extract. Some databases support automatic timestamp updates, meaning that the timestamp field's value is automatically updated when data in other fields of the table changes. Other capture methods include C. Full table comparison and D. Log comparison.

[0019] In the present invention, the big data platform module pushes the incrementally extracted data to the billing database. All operations performed on the billing database are insert operations. The billing database opens the binlog log. The binlog log is a database operation log that records the change operations on the billing database, such as: creating a database or table (create), insert operations (insert), update operations (update), delete operations (delete) and other operation types. The monitoring module only monitors the binlog of the insert type in the billing database. When an insert operation occurs in the billing main table, the MQ converted from the binlog is sent to the order transfer module. The order transfer module performs a billing database query, associates the billing main table with the billing sub-table to form a message, and sends the message MQ containing the data message content to itself; according to the content of the message MQ, the message data is converted into a billing standard order.

[0020] The structural diagram of the data processing device of the present invention is as follows Figure 1 As shown, the device is applied to a billing system, which includes:

[0021] The big data platform module 101 incrementally extracts the business main table and related business sub-tables of each business system and pushes them to the billing database 104 to form the corresponding billing main table and billing sub-tables;

[0022] The monitoring module 102 monitors the binary log of the insert type corresponding to the billing master table in the billing database, and sends the monitored binary log to the order transfer module through the message queue (MQ);

[0023] The order transfer module 103 obtains the business main table ID based on the billing main table in the MQ, and searches the billing sub-table associated with the billing main table from the billing database based on the business main table ID; composes the data of the billing main table and the associated billing sub-table into a message, and sends the message MQ containing the data message content to itself; and converts the message data into a billing standard order based on the content of the message MQ. The order transfer module of the present invention divides the MQ sent by the monitoring module into two parts and performs asynchronous processing. One part is only used for message splicing and assembly. By transferring the order through the message MQ, it can ensure that the MQ generated by the monitoring module is quickly consumed, does not cause backlog at the entry position, and is quickly diverted to the next MQ for asynchronous subsequent complex logic processing.

[0024] Among them, the billing main table and billing sub-table of the billing database also include a push time field (input date) for recording the time when the data table is pushed and inserted into the billing database; a billing system timestamp field (sys_ts) for recording when the data table is inserted and when the processing is completed; and a billing system processing status field (sys_opt_status) including two states: initial and processed.

[0025] The order transfer module 103 is also used to, after finding the billing sub-table associated with the billing main table, send the ID set consisting of the IDs of the billing main table and the associated billing sub-table to itself through the state change MQ; and update the billing system timestamp field and the billing system processing status field in the billing main table and the billing sub-table of the billing database according to the content of the state change MQ.

[0026] The order transfer module 103 is further configured to send the billing standard order to the standard order database 105 for storage.

[0027] Preferably,

[0028] The order transfer module 103 is also used to send the MQ received from the monitoring module to itself through a retry channel when the external interface cannot be adjusted or when the billing database search fails, so as to re-transfer the order when the network is restored.

[0029] The transfer module 103 is also used to write the message containing the data message content into the business exception table according to the billing system business verification rules when it receives the MQ sent by the monitoring module, or when it receives the message MQ containing the data message content itself, when it finds that the fields necessary for billing are missing.

[0030] The order transfer module 103 is also used to, after receiving the message MQ containing data message content, determine whether to write the message containing data message content into the duplicate exception table based on the billing status when it is determined that the business order number, business system source, document type, and transaction type are the same as the previous billing standard order content.

[0031] Based on the same inventive concept, the present invention also provides a data processing method, such as Figure 2 As shown, the method includes:

[0032] Step 21: The big data platform module incrementally extracts the main business table and associated business sub-tables for each business system and pushes them to the billing database to create the corresponding main billing table and sub-tables. This push can be scheduled daily at a fixed time. Currently, a staggered push schedule is used to reduce database pressure, with different types of documents pushed at different times.

[0033] The business master table of the business system includes the business master table ID, the data corresponding to the business master table ID at the business system creation time (create time), the data corresponding to the business master table ID at the business system update time (update time), the business order number corresponding to the business master table ID, the customer code, the billing field, and the billing fields such as volume, weight, origin, destination, etc.; the business sub-table of the business system includes multiple business sub-table IDs associated with each business master table ID, the data corresponding to each business sub-table ID at the business system creation time (create time), the data corresponding to each business sub-table ID at the business system update time (update time), and the billing status of the data corresponding to each business sub-table ID.

[0034] When the incrementally extracted business master table and associated business sub-tables of each business system are pushed to the billing database, the billing master table of the billing database corresponds to the business master table, and the billing sub-table corresponds to the business sub-table. The billing master table retains the field data of the business master table, and each piece of data is associated with a billing master table ID and the business master table ID. The retained fields include the business order number, customer code, billing field, create time, update time, etc. In addition, the push time field (input date) is used to record the time when the data table is pushed and inserted into the billing database; the billing system timestamp field (sys_ts) is used to record the insertion of the data table and the completion of processing; and the billing system processing status field (sys_opt_status), which includes two states: initial and processed.

[0035] Not every piece of data in the business subtable is pushed to the billing subtable. Whether to push each piece of data to the billing subtable in the billing database is determined based on the billing status of each piece of data corresponding to the business subtable ID. For each piece of data pushed to the billing subtable, the field data of the business subtable data is retained and a billing subtable ID is assigned to the data piece. This billing subtable ID corresponds one-to-one with the business subtable ID and is linked to the main business table ID. Retained fields include billing status, createtime, and update time. In addition, the input date field, sys_ts field, and sys_opt_status field have been added.

[0036] Step 22: The monitoring module monitors the insert-type binary log corresponding to the billing master table in the billing database, and sends the monitored binary log to the order transfer module through the message queue (MQ);

[0037] In this step, monitoring the insert-type binary log corresponding to the billing master table in the billing database includes: configuring the name of the billing database to be monitored, the name of the billing master table, and configuring to monitor only the insert-type binary log; and monitoring the insert-type binary log corresponding to the billing master table name in the billing database according to the configuration. It should be noted that in the present invention, the billing database enables binlog logging to record various modification operations on the billing database, but the monitoring module only monitors the insert-type binlog log.

[0038] Message queuing (MQ) is a method for application-to-application communication. Messaging refers to programs communicating by sending data in messages, rather than directly calling each other, which is typically used for technologies such as remote procedure calls. MQ offers high availability and performance, eliminating the need to worry about call failures or timeouts.

[0039] Step 23. The transfer order module obtains the business main table ID based on the billing main table in the MQ, and searches the billing sub-table associated with the billing main table from the billing database based on the business main table ID; composes the data of the billing main table and the associated billing sub-table into a message, and sends the message MQ containing the data message content to itself; and converts the message data into a billing standard order based on the content of the message MQ.

[0040] In this step, the message data is converted into a billing standard sheet according to the content of the message MQ, including: determining the business system source, document type, and transaction type on the billing standard sheet according to the billing master table name, and writing them into the billing standard sheet; writing the business order number, customer code, billing status, billing date, and billing fields in the message MQ into the billing standard sheet.

[0041] So far, the data processing method of the present invention is completed.

[0042] Furthermore, after executing step 23, the order transfer module sends the billing standard order to the standard order database for storage.

[0043] Furthermore, after finding the billing sub-table associated with the billing main table, the method also includes: passing the ID set consisting of the IDs of the billing main table and the associated billing sub-table to itself through the state change MQ; updating the billing system timestamp field and the billing system processing status field in the billing main table and the billing sub-table of the billing database according to the content of the state change MQ. Specifically, when the data table is inserted into the billing database, the value of the sys_ts field is the same as the value of the input date field, and the field values are both the time when the data table is inserted into the billing database, and the sys_opt_status field value is initial. When the order transfer module receives the state change MQ, it indicates that the order transfer is successful and the data table has been processed. At this time, the sys_ts field in the billing database is updated to the current time when the data table has been processed, and the value of the sys_opt_status field is updated to processed.

[0044] The main problem solved by the present invention is to complete the data docking of multiple upstreams without affecting the upstream, and convert the business system documents into standard documents (standard documents) required by the billing system. And through business rules such as data business verification and anti-duplicate verification, it is ensured that the number of documents required for billing by the billing system and the amount of business generated are accurate. And when the upstream changes, it can also adapt and change quickly. Therefore, the abnormal data of the present invention is also processed asynchronously by sending through MQ.

[0045] The first case: When the order transfer module cannot be called through the external interface or fails to search the billing database, the MQ received from the monitoring module will be sent to itself through the retry channel to re-transfer the order when the network is restored.

[0046] The second situation: When the transfer module receives the MQ sent by the monitoring module, or when it receives the message MQ containing the data message content, according to the business verification rules of the billing system, when it finds that the fields necessary for billing are missing, it writes the message containing the data message content into the business exception table.

[0047] The third situation: After receiving the message MQ containing data message content, the transfer module determines that the business order number, business system source, document type, and transaction type are the same as those of the previous billing standard order, and further determines whether to write the message containing the data message content into the duplicate exception table based on the billing status.

[0048] To clearly illustrate the present invention, specific scenarios are listed below for illustration.

[0049] Example 1

[0050] 1. The big data platform module incrementally extracts the main business table and related business sub-tables of each business system. In this embodiment, the main table extracted is the small-item pure matching order table, and the sub-table is the status flow table.

[0051] The main business table in the business system extracted at regular intervals every day is called lwb_main. The data in the main business table is shown in Table 1:

[0052]

[0053] Table 1

[0054] There are 4 data items in the business sub-table lwb_status that are associated with the business main table ID 140757507214515. The data in the business sub-table is shown in Table 2.

[0055]

[0056] Table 2

[0057] 2. The big data platform module pushes the data in Table 1 and Table 2 to the billing database according to the rules to form the corresponding billing main table and billing sub-table.

[0058] Table 1 is pushed to the billing system. The main billing table is named bi_lwb_main. The data in the main billing table is shown in Table 3:

[0059]

[0060] Table 3

[0061] The four pieces of data in Table 2 need to be pushed to the billing system's billing database based on the billing status field. According to business rules, in the fourth piece of data, the billing status field value of 1000 indicates successful delivery. This piece of data needs to be pushed to the billing database. The billing sub-table is named bi_lwb_status. The data in the billing sub-table is shown in Table 4:

[0062]

[0063] Table 4

[0064] 3. Monitoring module

[0065] 1) Configure the name of the billing database to be monitored as lbs_data, the name of the billing main table as bi_lwb_main, and configure the monitored binlog log type as insert;

[0066] 2) Monitor the insert type binlog log corresponding to the billing main table bi_lwb_main in the lbs_data billing database;

[0067] 3) The monitored binlog logs are spliced into JSON strings through Java programs or encapsulated into Java objects and sent to the order transfer module via MQ.

[0068] 4. Order transfer module

[0069] 1) The business main table ID 140757507214515 is obtained based on the billing main table in MQ. Since the billing main table bi_lwb_main is associated with the billing sub-table bi_lwb_status, and bi_lwb_main and bi_lwb_status are associated through the business main table ID 140757507214515, the billing sub-table bi_lwb_status associated with the billing main table bi_lwb_main is found in the billing database based on the business main table ID 140757507214515.

[0070] 2) The data of the billing main table bi_lwb_main and the associated billing sub-table bi_lwb_status are combined into a message, and the message MQ containing the data message content is sent to itself; the message data is converted into a billing standard sheet according to the content of the message MQ.

[0071] The topic of the message MQ is lbs_data_send_bilwb. In the same billing database, different billing master table names correspond to different topics and have different processing logic.

[0072] According to the billing master table bi_lwb_main, determine that the business system source on the billing standard document is the ECLP Canghai system, the document type is small part pure matching order, and the transaction type is small part pure matching ordinary.

[0073] Write the business order number (ECO140757507214515), customer code (EBU44180465480142019011401), billing status (delivered successfully), billing date (i.e., the createtime field value in the billing subtable (2019-01-15), and billing fields including volume and weight) in the MQ message into the billing standard form. The billing standard form is shown in Table 5:

[0074]

[0075] Table 5

[0076] The topic of the status change MQ is lbs_data_update_bilwb. When converted to a billing standard order, preferably, in an embodiment of the present invention, the ID set consisting of the billing main table ID13402170 and the billing sub-table ID13993313 is also sent to itself through the status change MQ; the billing system timestamp field and the billing system processing status field in the billing main table and the billing sub-table of the billing database are updated according to the content of the status change MQ.

[0077] For another example, in another embodiment, the business main table is named so_main, and the billing main table is named bi_so_main. In the same billing database, different billing main table names correspond to different topics. The topic of the message MQ is lbs_data_send_biso, and the topic of the status change MQ is lbs_data_update_biso. Based on the billing main table bi_so_main, it can be determined that the business system source on the billing standard sheet is the ECLP Canghai system, the document type is a sales order, and the transaction type is an ordinary sales order. Furthermore, if there are multiple types of transactions, the business main table names correspond one-to-one to the transaction type types, and then the billing main table names correspond one-to-one to the transaction types. Since different billing main table names correspond to different topics, the transaction type, document type, and business system source are still determined by the billing main table name.

[0078] Example 2

[0079] The data processing solution of the present invention can also re-push data when data anomalies such as loss or error occur, and can also prevent duplication when billing standard singles are repeated.

[0080] In the first case, the order transfer module needs to call an external interface and distinguish the transaction type based on the returned value. If the network times out, the external interface cannot be called; or, when searching the billing database to obtain the billing sub-table associated with the billing main table, a network timeout may occur. At this time, the MQ received from the monitoring module is sent to itself through the retry channel so that the order can be transferred again when the network is restored.

[0081] In the second case, when the transfer module receives the MQ sent by the monitoring module, or when it receives the message MQ containing the data message content, according to the billing system service verification rules, if the weight field value of the billing field is found to be empty, it is considered a service exception and cannot be transferred to the billing standard order. Therefore, the message containing the data message content is written into the service exception table. The service exception table in this embodiment is shown in Table 6:

[0082]

[0083] Table 6

[0084] In the third case, after receiving the message MQ containing the data message content, the transfer module needs to further judge the billing status when it determines that the business order number, business system source, document type, and transaction type are the same as those of the previous billing standard order. Assuming that the billing status of the previous billing standard order is delivery, and the billing status of the current billing standard order is rejection, delivery and rejection are two billing states that cannot coexist. Therefore, it can be determined that the current billing standard order is a duplicate and needs to be written into the duplicate exception table. The duplicate exception table in this embodiment is shown in Table 7:

[0085]

[0086] Table 7

[0087] It should be noted that Tables 1 to 7 are only examples of specific embodiments. The tables show necessary fields, and different fields may be displayed for specific applications.

[0088] The modules of the above embodiments may be integrated into one body or deployed separately; they may be combined into one module or further split into multiple sub-modules.

[0089] In addition, an electronic device is also provided in the embodiment of the present application, the structural diagram of which is shown in FIG. Figure 3 As shown, it includes a memory 301, a processor 302 and a computer program stored in the memory and capable of running on the processor, and when the processor executes the program, the steps of the target positioning method of sequence data are implemented.

[0090] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the target positioning method of the sequence data are implemented.

[0091] The data processing method and device of the present invention will bring the following benefits:

[0092] 1. Monitoring binlog logs only requires monitoring inserts into the billing master table. Because the main information of a document is stored in the billing master table and is linked to the billing sub-tables, monitoring the billing master table information ensures data integrity. This is equivalent to monitoring the full amount of inserts to generate binlogs, which can reduce the number of MQs, save resources, and alleviate system pressure.

[0093] Second, the order transfer module splits the MQ messages sent by the monitoring module into two parts for asynchronous processing. One part performs only message assembly and transfers the order via the message MQ; the other part modifies the status via the status change MQ. Compared to the order transfer module directly assembling the message, modifying the status, and transferring the order, this ensures that the MQ messages generated by the monitoring module are consumed quickly, preventing backlogs at the entry point and allowing them to be quickly diverted to the next MQ for asynchronous subsequent complex logic processing.

[0094] Third, the big data platform module pushes orders to the billing system's database at a fixed time each day. This relatively short period of time generates a large number of binlongs and, consequently, a large number of MQ messages. However, since the billing system's MQ threads are fixed, the number of messages that MQ can process at any one time is also fixed. Therefore, the billing system can manage pressure through MQ caching, resulting in only a short-term MQ backlog.

[0095] 4. Abnormal data is also processed asynchronously by sending it through MQ. Through a scheduling framework such as (clover), this data can be reprocessed later.

[0096] 5. Binlog log monitoring allows for integration with business systems without the need for direct interaction with upstream business systems.

[0097] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: The method includes: The big data platform module incrementally extracts the main business table and related business sub-tables of each business system, and pushes them to the billing database to form the corresponding billing main table and billing sub-tables; The monitoring module monitors the insert-type binary logs corresponding to the billing master table in the billing database and sends the monitored binary logs to the order transfer module through the message queue; The transfer module obtains the service master table ID from the billing master table in the message queue, searches the billing database for the billing sub-table associated with the billing master table based on the service master table ID, assembles the data from the billing master table and the associated billing sub-table into a message, and sends the message queue containing the data message content to itself; and converts the message data into a standard billing order based on the content of the message queue; the billing master table and the billing sub-table are associated via the service master table ID, and the billing master table and the associated billing sub-table are different; When the order transfer module receives the message queue sent by the monitoring module, or when it receives the message message queue containing data message content, it asynchronously writes the message containing the data message content into the business exception table according to the business verification rules of the billing system when it finds that the necessary fields for billing are missing; The billing main table and billing sub-table of the billing database further include a push time field for recording the time when the data table is pushed and inserted into the billing database; a billing system timestamp field for recording the time when the data table is inserted and when the data is processed; and a billing system processing status field including two states: initial and processed. After finding the billing sub-table associated with the billing main table, the method further includes: The order transfer module asynchronously sends the ID set consisting of the IDs of the billing main table and the associated billing sub-table to itself through the status change message queue; updates the billing system timestamp field and the billing system processing status field in the billing main table and the billing sub-table of the billing database according to the content of the status change message queue; when the order transfer module receives the status change message queue, it indicates that the order transfer is successful or the data table has been processed.

2. The method according to claim 1, wherein The method further includes: sending the billing standard form to a standard form database for storage.

3. The method according to claim 1, wherein The business main table of the business system includes a business main table ID, the data corresponding to the business main table ID at the business system creation time, the data corresponding to the business main table ID at the business system update time, the business order number, customer code, and billing field corresponding to the business main table ID; the business sub-table of the business system includes multiple business sub-table IDs associated with each business main table ID, the data corresponding to each business sub-table ID at the business system creation time, the data corresponding to each business sub-table ID at the business system update time, and the billing status of the data corresponding to each business sub-table ID.

4. The method according to claim 3, wherein The pushing to the billing database to form a corresponding billing sub-table includes: Determine whether to push each piece of data to the billing subtable of the billing database based on the billing status of each piece of data corresponding to the business subtable ID; When the data is pushed to the billing sub-table of the billing database, a billing sub-table ID is assigned to the data. The billing sub-table ID corresponds to the business sub-table ID one-to-one and is associated with the business main table ID.

5. The method according to claim 1, wherein The binary logs of the insert type corresponding to the billing master table in the monitoring and billing database include: Configure the name of the billing database to be monitored, the name of the billing master table, and configure it to monitor only insert-type binary logs; Monitor the binary log of the insert type corresponding to the main billing table name in the billing database according to the configuration.

6. The method according to claim 4, wherein Converting message data into a billing standard form based on the message queue content includes: Determine the business system source, document type, and transaction type on the billing standard form based on the billing master table name, and write them into the billing standard form. Write the business order number, customer code, billing status, billing date, and billing fields in the message queue into the billing standard order.

7. The method according to claim 1, wherein When the order transfer module cannot be called through the external interface or fails to search the billing database, it will send the message queue received from the monitoring module to itself through the retry channel to re-transfer the order when the network is restored.

8. The method according to claim 1, wherein After receiving the message queue containing data message content, the transfer module determines that the business order number, business system source, document type, and transaction type are the same as those of the previous billing standard order, and further determines whether to write the message containing data message content into the duplicate exception table based on the billing status.

9. A data processing device, characterized in that: The device is applied to a billing system, which includes: The big data platform module is used to incrementally extract the main business table and related business sub-tables of each business system and push them to the billing database to form the corresponding billing main table and billing sub-tables; The monitoring module is used to monitor the binary logs of the insert type corresponding to the billing master table in the billing database and send the monitored binary logs to the order transfer module through the message queue; The transfer module is used to obtain the service master table ID from the billing master table in the message queue, search the billing sub-table associated with the billing master table from the billing database based on the service master table ID; combine the data of the billing master table and the associated billing sub-table into a message, and send a message message queue containing the data message content to itself; and convert the message data into a billing standard form based on the content of the message message queue; the billing master table and the billing sub-table are associated via the service master table ID, and the billing master table and the associated billing sub-table are different; The transfer module is also used to asynchronously write the message containing the data message content into the business exception table according to the billing system business verification rules when receiving the message queue sent by the monitoring module, or when receiving the message message queue containing the data message content itself; The billing main table and billing sub-table of the billing database further include a push time field for recording the time when the data table is pushed and inserted into the billing database; a billing system timestamp field for recording the time when the data table is inserted and when the data is processed; and a billing system processing status field including two states: initial and processed. The order transfer module is also used to asynchronously send the ID set consisting of the IDs of the billing main table and the associated billing sub-table to itself through a status change message queue after finding the billing sub-table associated with the billing main table; update the billing system timestamp field and the billing system processing status field in the billing main table and the billing sub-table of the billing database according to the content of the status change message queue; when the order transfer module receives the status change message queue, it indicates that the order transfer is successful or the data table has been processed.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images