Hybrid data transmission method and system, storage medium and electronic equipment

Abstract

The invention discloses a mixed data transmission method and system, a storage medium and electronic equipment, relates to the technical field of data processing, and aims to complete mixed data receiving and sending of batch data and real-time data by integrating batch and real-time data transmission interfaces and using a uniform interface so as to avoid respective receiving and sending of batch stream data and improve the data transmission efficiency. The ambiguity problem generated by using different interfaces is solved, and the consistency, accuracy and timeliness of mixed data, such as real-time data and batch data in the transmission process are improved. Besides, transmission and retransmission of historical batch data can be realized by adjusting the data acquisition configuration table, and the data acquisition configuration table records related information of data acquisition and transmission, so that transmission strategies of mixed data, including processing of historical data, can be flexibly controlled by modifying parameters and conditions in the data acquisition configuration table. And the flexibility of the real-time data and the batch data in the transmission process is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a method, system, storage medium, and electronic device for transmitting mixed data. Background Technology

[0002] In the field of data processing in the software industry, as business becomes more complex and data volume continues to grow, the requirements for data transmission are also increasing.

[0003] In existing technologies, real-time data transmission and batch data transmission employ different send / receive interfaces and processing methods. Real-time data emphasizes immediacy and is typically acquired from real-time data sources such as message queues and sent directly. Batch data, on the other hand, focuses on the centralized processing of large-scale data, acquiring it from batch data sources such as database tables, exporting it into files according to certain formats and rules, and then sending it.

[0004] The aforementioned separate data transmission and reception method has the following drawbacks: On the one hand, for common data, the separate transmission and reception of data may lead to ambiguity in downstream data reception and processing, affecting data consistency and accuracy; on the other hand, it is not conducive to the rapid flow of batch-generated data and makes it difficult to meet the high requirements of business for data timeliness.

[0005] Therefore, how to improve the consistency, accuracy and timeliness of mixed data, such as real-time data and batch data, during the transmission process is an urgent problem to be solved in this application. Summary of the Invention

[0006] In view of this, this application discloses a method, system, storage medium and electronic device for transmitting mixed data, aiming to improve the consistency, accuracy and timeliness of mixed data, such as real-time data and batch data, during the transmission process.

[0007] To achieve the above objectives, the disclosed technical solution is as follows:

[0008] The first aspect of this application discloses a method for transmitting mixed data, the method comprising:

[0009] When data to be processed is acquired, the data source of the data to be processed is distinguished according to the data acquisition configuration table;

[0010] If the data to be processed originates from a database table, batch data is retrieved from the database table according to the data acquisition control table;

[0011] If the batch data meets the batch sending conditions, multiple data to be sent are obtained;

[0012] The multiple data items to be sent are sent in batches to a pre-configured target queue through a batch task sending process and a dynamic paging method; wherein, the dynamic paging method is a dynamic paging method that takes into account network quality, system load and historical data sending conditions.

[0013] If the data to be processed originates from a message queue, real-time data is retrieved from the message queue in real time according to the message offset, and the real-time data is sent to a pre-configured target queue through a real-time task sending process.

[0014] Preferably, when data to be processed is acquired, distinguishing the data source of the data to be processed according to the data acquisition configuration table includes:

[0015] When data to be processed is acquired, the data source is distinguished according to the acquisition type in the data acquisition configuration table; wherein, the acquisition type includes at least batch data and real-time data.

[0016] Preferably, if the data to be processed originates from a database table, retrieving batch data from the database table based on batch status information includes:

[0017] If the data to be processed originates from a database table, batch data is obtained from the database table according to the batch status information in the data acquisition control table; wherein, the batch status information is determined by the batch field value, and the batch field value includes at least the date and batch identifier.

[0018] Preferably, if the batch data meets the batch sending conditions, multiple data to be sent are obtained, including:

[0019] If the batch data meets the initialization of the sending batch, it is determined that the batch data meets the batch sending conditions, and multiple data to be sent are obtained.

[0020] The process of sending batch initialization includes:

[0021] Read multiple records of data with batch data type from the data acquisition configuration table;

[0022] For multiple pieces of information, determine whether the time interval between the current processing time and the previous processing time of the information is greater than a preset collection interval;

[0023] If the time interval is less than or equal to the preset collection interval, perform time interval determination processing on the next piece of information;

[0024] If the time interval is greater than the preset collection interval, query the collection table name for the set of batch field names and the set of batch field values ​​that meet the collection conditions;

[0025] Use the set of batch field names and the set of batch field values ​​as combined conditions to query the data acquisition control table;

[0026] If there is no matching record in the data acquisition control table, add the acquisition table name, batch field value data and batch sending status to the initialization state corresponding to the combined conditions to the data acquisition control table.

[0027] If a matching record exists in the data acquisition control table, it is determined that the current batch sending task in the batch data has completed the batch initialization, and the next batch sending task is executed to complete the batch initialization, until the batch initialization operation of all batch sending tasks in the batch data is completed.

[0028] Preferably, the step of sending the multiple data to be sent in batches to a pre-configured target queue through a batch task sending process and dynamic paging includes:

[0029] Query the records in the data acquisition control table whose batch sending status is initialized;

[0030] If the query result is empty, end the current loop.

[0031] If the query result is not empty and there are multiple initialized records, for the multiple initialized records, the collection table name in the collection control table and the batch field value in the corresponding batch condition are converted into a query script according to the records; wherein, the record represents a collection data table and the corresponding batch condition;

[0032] The query script is executed to obtain the data information to be sent from the query data collection control table through dynamic pagination based on comprehensive network quality, system load, and historical data transmission status.

[0033] The multiple data messages to be sent are converted one by one into a preset standard format, and the converted data messages are sent to the configured target queue through the batch task sending process, thereby completing the process of sending the multiple data messages to the pre-configured target queue in batches through the batch task sending process.

[0034] Preferred options also include:

[0035] If an anomaly is detected during the transmission of the batch-initialized data after format conversion to the configured target queue, and the transmission fails after a preset number of retries, the unsuccessfully transmitted batch-initialized data is written into the data retransmission information table, and the task self-healing repair process is started to repair the abnormal data.

[0036] Preferred options also include:

[0037] When each page of data is processed, the number of successfully sent records and the number of failed records in the data acquisition control table are updated. When the last page is processed, the batch sending status is updated, and the process returns to query the data acquisition control table for records with the batch sending status set to initial. This process continues until all records with the batch sending status set to initial in the data acquisition control table have been processed.

[0038] A second aspect of this application discloses a hybrid data transmission system, the system comprising:

[0039] The differentiation unit is used to differentiate the data source of the data to be processed according to the data acquisition configuration table when the data to be processed is acquired.

[0040] The first acquisition unit is used to acquire batch data from the database table according to the data acquisition control table if the data to be processed comes from the database table.

[0041] The second acquisition unit is used to obtain multiple data to be sent if the batch data meets the batch sending conditions;

[0042] A batch sending unit is used to send the multiple data to be sent to a pre-configured target queue in batches through a batch task sending process;

[0043] The acquisition and sending unit is used to acquire real-time data from the message queue in real time according to the message offset if the data to be processed comes from the message queue, and send the real-time data to the pre-configured target queue through the real-time task sending process.

[0044] A third aspect of this application discloses a storage medium including storage instructions, wherein, when the instructions are executed, the device in which the storage medium is located is controlled to perform a mixed data transmission method as described in any one of the first aspects.

[0045] The fourth aspect of this application discloses an electronic device including a memory and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors using the mixed data transmission method as described in any one of the first aspects.

[0046] As can be seen from the above technical solutions, this application discloses a method, system, storage medium, and electronic device for transmitting mixed data. When data to be processed is acquired, the data source of the data to be processed is distinguished according to the data acquisition configuration table. If the data to be processed comes from a database table, batch data is obtained from the database table according to the data acquisition control table. If the batch data meets the batch sending conditions, multiple data to be sent are obtained and multiple data to be sent in batches are sent to the pre-configured target queue through the batch task sending process. If the data to be processed comes from a message queue, real-time data is obtained from the message queue in real time according to the message offset and the real-time data is sent to the pre-configured target queue through the real-time task sending process.

[0047] The beneficial effects of this application are as follows: By integrating batch and real-time data transmission interfaces, a unified interface is used to complete the mixed data reception and transmission of batch and real-time data, avoiding the ambiguity caused by separate batch and stream data transmission using different interfaces. This improves the consistency, accuracy, and timeliness of mixed data, such as real-time data and batch data, during transmission. Furthermore, by adjusting the data acquisition configuration table, the transmission and retransmission of historical batch data can be achieved. Since the data acquisition configuration table records relevant information for data acquisition and transmission, modifying the parameters and conditions in the data acquisition configuration table allows for flexible control of the mixed data transmission strategy, including the processing of historical data, thus improving the flexibility of real-time and batch data transmission. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating a method for transmitting mixed data disclosed in an embodiment of this application;

[0050] Figure 2 This is a schematic diagram of the transmission apparatus for obtaining mixed data disclosed in the embodiments of this application;

[0051] Figure 3 This is a flowchart of the batch task initialization component disclosed in the embodiments of this application;

[0052] Figure 4 This is a flowchart of the batch task sending component disclosed in the embodiments of this application;

[0053] Figure 5 This is a flowchart of the real-time task sending component disclosed in the embodiments of this application;

[0054] Figure 6 A flowchart of the data resending component disclosed in the embodiments of this application;

[0055] Figure 7 This is a schematic diagram of the structure of a hybrid data transmission system disclosed in an embodiment of this application;

[0056] Figure 8 This is a schematic diagram of the structure of the electronic device disclosed in the embodiments of this application. Detailed Implementation

[0057] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0058] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0059] As can be seen from the background technology, the separate data transmission and reception method has the disadvantages of causing ambiguity in downstream data reception and processing, affecting data consistency and accuracy, and hindering the rapid flow of batch-generated data, making it difficult to meet the high requirements of business for data timeliness.

[0060] To address the aforementioned issues, this application discloses a method, system, storage medium, and electronic device for transmitting mixed data. By integrating batch and real-time data transmission interfaces, a unified interface is used to complete the reception and transmission of mixed batch and real-time data. This avoids the ambiguity caused by separate transmission and reception of batch and stream data using different interfaces, improving the consistency, accuracy, and timeliness of mixed data, such as real-time data and batch data, during transmission. Furthermore, by adjusting the data acquisition configuration table, the transmission and retransmission of historical batch data can be achieved. Since the data acquisition configuration table records relevant information for data acquisition and transmission, modifying the parameters and conditions in the data acquisition configuration table allows for flexible control of the mixed data transmission strategy, including the processing of historical data, thus improving the flexibility of real-time and batch data transmission. Specific implementation methods are described in detail in the following embodiments.

[0061] refer to Figure 1 The illustration shows a method for transmitting mixed data according to an embodiment of this application. This method is applicable to the following mixed data transmission apparatus and mainly includes the following steps:

[0062] S101: When data to be processed is acquired, the data source of the data to be processed is distinguished according to the data acquisition configuration table (CM_MSG_CFG).

[0063] In step S101, when data to be processed is acquired, the data source is distinguished according to the data acquisition type configuration information in the data acquisition configuration table; where the data to be processed is data whose data source has not been distinguished. The acquisition type includes at least batch data and real-time data.

[0064] Batch data comes from database tables, while streaming data comes from message queues.

[0065] Batch data refers to data stored in a database table in batches. It is usually a collection of data accumulated over a period of time and is processed in whole batches.

[0066] Streaming data is data that is continuously generated and transmitted in real time, such as data collected by sensors in real time and log data of system operation. It is characterized by fast data generation speed, variable data volume and continuous flow.

[0067] For example, if the data type to be processed is determined to be a batch data type based on the data type in the data acquisition configuration table, then the data to be processed is determined to originate from a database table; if the data type to be processed is determined to be a real-time data type based on the data acquisition configuration table, then the data to be processed is determined to originate from a message queue.

[0068] Message queues are tools used to pass messages between different programs or components. They enable asynchronous data transmission and buffering, ensuring that data is not lost during sending and receiving, and are often used for real-time data transmission.

[0069] The specific data collection configuration table is shown in Table 1.

[0070] Table 1

[0071] Serial Number English name Chinese name Notes 1 IND_SRC_TYP Collection type 1-Batch data types 2-Real-time data types 2 NAM_SRC_TABLE Collection table name 3 TXT_SRC_WHERE Data collection conditions The extended conditions for reading data from a table are typically configured so that the data date is greater than or equal to the current date - N days, meaning data generated within N days of collection. 4 NAM_SRC_BATKEY Batch field name Multiple fields can be configured, such as date + batch identifier. 5 VAL_SRC_INTERVAL Acquisition interval 6 IND_PAGE_ISNO Pagination indicator 7 CNT_BASE_PAGE_SIZE Data volume per page Initial data volume per page 8 CNT_PAGE_SIZE_MIN Minimum value of data per page 9 CNT_PAGE_SIZE_MAX Maximum data per page 10 VAL_SEND_ORDER Prioritization 11 NAM_SEND_TOPIC Send target name 12 TH_FAIL_CNT Threshold for the number of failed transmissions 13 TH_FAIL_CNT_MIN Minimum threshold for the number of failed transmissions 14 TH_FAIL_CNT_MAX Maximum threshold for number of failed transmissions

[0072] S102: If the data to be processed comes from a database table, retrieve the batch data from the database table according to the data acquisition control table (CM_MSG_CTL).

[0073] In S102, if the data to be processed comes from a database table, batch data is obtained from the database table according to the batch status information in the data acquisition control table; wherein, the batch status information is determined by the batch field value, and the batch field value includes at least the date and batch identifier.

[0074] The contents of the specific data acquisition control table are shown in Table 2.

[0075] Table 2

[0076] Serial Number English name Chinese name Notes 1 NAM_SRC_TABLE Collection table name 2 VAL_SRC_BATKEY Batch field value Batch identifiers for batch processing are stored, typically a data date or a combination of a data date and a specific field, such as DATE='20241231', BAT='A01'. 3 IND_BAT_STATE Batch sending status 0 - Initialization, 1 - Sending, 2 - All sent successfully, 3 - Sending successful, some parts failed. 4 TME_BAT_CREAT Creation time Batch creation time 5 CNT_BAT_TOTAL Total number of batches This batch of data table records the number of records. 6 TME_BAT_BEGIN Send start time 7 TME_BAT_END Sending end time 8 CNT_SND_SUCC Successful Send Count Number of records sent 9 CNT_SND_FAIL Send failure count

[0077] For example, the batch identifier of the result of running program A001 is batch A001, and the batch identifier of the result of running program A002 is batch A002. Batch A001 and batch A002 do not affect each other.

[0078] S103: If the batch data meets the batch sending conditions, multiple data to be sent are obtained.

[0079] It should be noted that batch data meets the batch sending conditions, that is, batches that meet the initialization conditions.

[0080] If the batch data meets the initialization requirements for sending the batch, it is determined that the batch data meets the batch sending conditions, and multiple data to be sent are obtained.

[0081] The process of sending batch initialization is shown in A1-A7.

[0082] A1: Read multiple records of data in the data collection configuration table that are of batch data type.

[0083] In A1, read the information of the data collection type 1-batch data from the data collection configuration table to obtain the query result list, and process each of the multiple pieces of information in the query result list as follows.

[0084] A2: For multiple pieces of information, determine whether the time interval between the current processing time and the previous processing time is greater than the preset collection interval.

[0085] The preset collection interval can be configured according to the actual situation, and this application does not impose specific limitations.

[0086] A3: If the time interval is less than or equal to the preset collection interval, perform time interval determination processing on the next piece of information.

[0087] In A3, if the time interval is less than or equal to the preset collection interval, the loop will exit and the time interval determination process will be performed on the next piece of information.

[0088] A4: If the time interval is greater than the preset collection interval, query the collection table for the set of batch field names and the set of batch field values ​​that meet the collection conditions.

[0089] The collection conditions are the extended conditions for reading the data table in the data collection configuration table. They are generally configured as the data date being greater than or equal to the current date - N days, meaning that data will be generated within N days of collection.

[0090] Based on the data collection table name and collection conditions, query the set of batch field names and the set of batch field values ​​that meet the collection conditions in the data collection table name.

[0091] A5: Use the set of batch field names and the set of batch field values ​​as combined conditions to query the data collection control table.

[0092] A6: If there is no matching record in the data acquisition control table, add the acquisition table name, batch field value data and batch sending status corresponding to the combined conditions to the data acquisition control table as the initial state.

[0093] In A6, if there are no matching records in the data acquisition control table, add the corresponding acquisition table name and batch field value data, and the batch sending status is 0-initialization.

[0094] A7: If a matching record exists in the data acquisition control table, determine that the current batch sending task in the batch data has completed the batch initialization, and execute the next batch sending task to complete the batch initialization, until the batch initialization operation of all batch sending tasks in the batch data is completed.

[0095] In A7, if a matching record exists in the data acquisition control table, it means that the batch sending task has been initialized and the current loop processing will be terminated.

[0096] When the current message is completed, the preset collection interval is paused, and the monitoring and scanning of the next message begins, which means returning to the operation of reading the data collection configuration table.

[0097] S104: Multiple data items to be sent are batch-sent to a pre-configured target queue using a batch task sending process and dynamic paging. The dynamic paging method is a dynamic paging method that considers network quality, system load, and historical data sending data. The batch task sending process is started, and the following steps B1-B5 are performed repeatedly until the task of batch-sending multiple data items to be sent to the pre-configured target queue is completed.

[0098] Specifically, the process of sending multiple data items to a pre-configured target queue in batches through a batch task sending process is shown in B1-B5.

[0099] B1: Query the records in the data acquisition control table whose batch sending status is initialized.

[0100] B2: If the query result is empty, end the current loop.

[0101] B3: If the query result is not empty and there are multiple initialized records, for the multiple initialized records, convert the collection table name in the collection control table and the batch field value in the corresponding batch condition into a query script based on the records; where a record represents a collection data table and the corresponding batch condition.

[0102] In B3, if the query result is not empty, multiple records are processed one by one. Each record represents a data collection table and the corresponding batch conditions. The data collection control table's collection table name and batch field value are converted into a query script in Structured Query Language (SQL) format.

[0103] The query script can also be in other formats, and this application does not impose any specific restrictions.

[0104] B4: Execute the query script to retrieve the data to be sent from the query data collection control table in pagination.

[0105] The method of dynamically calculating and paginating the transmitted data by taking into account network quality, system load, and historical data transmission status is used to obtain the data information to be sent from the query data collection control table.

[0106] The paging settings that impact performance are automatically adjusted based on network conditions, system load, and historical statistics. An automatic performance optimization component automatically adjusts the paging size to optimize system performance, setting safe boundary ranges for the key parameter PageSize, which represents the amount of data per page in batch / stream transmission tasks.

[0107] Different data sources are configured with different security boundary ranges.

[0108] B5: Convert multiple data messages to be sent one by one into a preset standard format, and send the converted data messages to the configured target queue through the batch task sending process, so as to complete the process of sending multiple data messages to the pre-configured target queue in batches through the batch task sending process.

[0109] If an anomaly is detected during the transmission of the batch-initialized data after format conversion to the configured target queue, and the transmission is retried a preset number of times without success, the unsuccessfully transmitted batch-initialized data will be written into the data retransmission information table (CM_MSG_REG), and the task self-healing repair process will be started to repair the abnormal data.

[0110] When each page of data is processed, the number of successfully sent records and the number of failed records in the data acquisition control table are updated. When the last page is processed, the batch sending status is updated, and the process returns to query the data acquisition control table for records with the batch sending status set to initial. This process continues until all records with the batch sending status set to initial in the data acquisition control table have been processed.

[0111] The contents of the specific data reissue information table are shown in Table 3.

[0112] Table 3

[0113] Serial Number English name Chinese name Notes 1 MSG_ID Message ID 2 NAM_SRC_TABLE Collection table name 3 VAL_SRC_BATKEY Batch field value 4 TOPIC_ID Send queue topic ID 5 MSG_SEND_TIME Message sending and writing time 6 MSG_STATUS Message sending status 7 MSG_TXT Message content 8 MSG_FAIL_COUNT Number of message sending failures 9 MSG_FAIL_REASON Reasons for message sending failure 10 MSG_SENDED_TIME Message sending completion time

[0114] S105: If the data to be processed comes from the message queue, real-time data is obtained from the message queue in real time according to the message offset, and the real-time data is sent to the pre-configured target queue through the real-time task sending process.

[0115] The message offset utilizes Kafka's features. The message offset is stored on the server, and each consumer's offset is saved and maintained for its own thread group ID.

[0116] The configuration table in this solution can be replaced by a configuration file, etc. The data format sent can be replaced by XML or other formats.

[0117] In the field of data processing within the software industry, the transmission of real-time data and batch data typically employs different interfaces and processing methods. Taking banking risk control as an example, different risk models process risk clues using either streaming or batch methods based on the applicable business scenario and data source. Regardless of the method used, the common goal is to transfer the data to the next stage for verification and confirmation as early as possible. This application aims to integrate batch and real-time data transmission interfaces to achieve the sending and receiving of mixed batch and streaming data, improving data flow efficiency while avoiding the ambiguity caused by separate sending and receiving of batch and streaming data using different interfaces.

[0118] This application is particularly suitable for scenarios in the banking risk control field that require the simultaneous transmission of historical batch and real-time data. Data sources are differentiated through configuration information. For batch data, the data source is a database table; batch data is obtained according to batch status information, standardized, and then sent to a designated message queue. For streaming data, the data source is a message queue; real-time data is obtained according to message offsets, standardized, and then sent to a designated message queue. The data receiving component uses a unified interface to complete data reception and processing, avoiding the drawbacks of using different interfaces for batch and streaming data. Performance-critical pagination is automatically adjusted based on network conditions, system load, and historical statistics, supporting a triple protection mechanism of task self-healing, data retransmission, and alarm notification.

[0119] The beneficial effects of this application's embodiments are as follows: By integrating batch and real-time data transmission interfaces, a unified interface is used to complete the mixed data reception and transmission of batch and real-time data, avoiding the ambiguity caused by separate batch and stream data transmission using different interfaces. This improves the consistency, accuracy, and timeliness of mixed data, such as real-time data and batch data, during transmission. Furthermore, by adjusting the data acquisition configuration table, the transmission and retransmission of historical batch data can be achieved. Since the data acquisition configuration table records relevant information for data acquisition and transmission, modifying the parameters and conditions in the data acquisition configuration table allows for flexible control of the mixed data transmission strategy, including the processing of historical data, thus improving the flexibility of real-time and batch data transmission.

[0120] refer to Figure 2 The diagram shown is a schematic of a hybrid data transmission device disclosed in an embodiment of this application. The hybrid data transmission device includes a batch task initialization component, a batch task sending component, a real-time task sending component, a data retransmission component, a performance automatic optimization component, a task self-healing and repair component, and a data receiving component.

[0121] The mixed data transmission device uses configuration information to distinguish data sources, obtains batch data based on batch status information, processes it in a standardized format, and sends it to the configured message queue; it also obtains data according to message offset, processes it in a standardized format, and sends it to the configured message queue.

[0122] For batch data transmission, the main components include a batch task initialization component and a batch task transmission component. The batch task initialization component monitors the data table to be transmitted. When it detects data that meets the batch transmission conditions, it initiates the batch task initialization process to initialize the transmission batches. The batch transmission initialization process is as follows: Figure 3 As shown. Figure 3 A flowchart of the batch task initialization component is shown.

[0123] Batch task initialization component:

[0124] Start the batch task initialization process, and perform the following steps in a loop:

[0125] Step 1: Read the information with collection type 1-batch data from the data collection configuration table (CM_MSG_CFG) to obtain the query result list, and process each of the multiple pieces of information in the query result list as follows.

[0126] Step 2: Process the read results one by one, starting with the i-th result;

[0127] Step 2.1: For multiple messages, determine whether the time interval between the current processing time and the previous processing time is greater than the preset collection interval. If the time interval is less than or equal to the preset collection interval, exit the current loop. If the time interval is greater than the preset collection interval, proceed to step 2.2.

[0128] Step 2.2: Based on the data collection table name and collection conditions, query the set of batch field names and the set of batch field values ​​that meet the collection conditions in the data collection table name.

[0129] Step 2.3: Use the obtained batch field name, batch field value set, and acquisition table name as combined conditions to query the data acquisition control table (CM_MSG_CTL). If there is no matching record in the data acquisition control table, add the corresponding acquisition table name and batch field value data, and set the batch sending status to 0-initialization. If there is already a matching record in the data acquisition control table, it means that this batch sending task has completed initialization, and the current loop process is terminated.

[0130] Step 3: After completing Step 2, pause the preset collection interval and proceed to the next round of monitoring and scanning, that is, return to the operation of reading the data collection configuration table.

[0131] Batch task sending component:

[0132] The batch task sending component is used to obtain batch task information and start the batch task sending process to read and send data. The batch task sending process is started and repeated cyclically. Figure 4 As shown. Figure 4 A flowchart of the batch task sending component is shown.

[0133] Step 1: Read the records in the data acquisition control table (CM_MSG_CTL) where the batch transmission status is 0-initialization.

[0134] Step 2: If the query result is empty, end the current loop. If the query result is not empty and there are multiple initialized records, process each record one by one. Each record represents a data collection table and the corresponding batch conditions. Convert the data collection control table name and batch field values ​​into an SQL query script.

[0135] Step 3: Execute the query script to retrieve the data to be sent from the data collection table in pages.

[0136] Step 4: Convert each retrieved data item into standard JSON format and send it to the configured message queue. JSON is a lightweight data exchange format that is easy for humans to read and write, as well as easy for machines to parse and generate. It is commonly used for data storage and transmission, and can clearly represent the structure and content of data.

[0137] Step 4.1: If an error occurs during the data transmission process and the transmission still fails after multiple retries, write this information to the data retransmission information table.

[0138] Step 5: After processing each page of data, update the number of successful and failed transmission records in the data acquisition control table (CM_MSG_CTL). When the last page is processed, update the batch transmission status at the same time.

[0139] Step 6: This cycle ends. After pausing for the set time, the next cycle will begin.

[0140] The task sending component is used to process streaming data. After starting the real-time task sending process, it performs the following processing:

[0141] Step 1: For real-time data, read the data acquisition configuration table where the acquisition type is "2-Real-time Data".

[0142] Step 2: Read the data from the message queue according to the collection table name.

[0143] Step 3: Process the data into standard JSON format and send it to the configured destination name queue. After sending, continue to step 2 to read the next piece of data.

[0144] Step 3.1 If an error occurs during processing, write the error information into the data resend information table.

[0145] Real-time task sending component:

[0146] The real-time task sending component is used to initiate the real-time task sending process and process streaming data, specifically as follows: Figure 5 As shown.

[0147] Initiate the real-time task sending process and perform the following steps:

[0148] Step 1: For real-time data, read the data acquisition configuration table where the acquisition type is "2-Real-time Data".

[0149] Step 2: Read the data from the message queue according to the collection table name.

[0150] Step 3: Process the data into standard JSON format and send it to the pre-configured destination name queue. After sending, continue to step 2 to read the next piece of data.

[0151] Step 3.1: If an error occurs during processing, write the error information into the data resend information table.

[0152] Data retransmission component:

[0153] The process for data retransmission components is as follows: Figure 6As shown. The data retransmission component is used to initiate the data retransmission process and performs the following steps:

[0154] Step 1: Query the pending messages in the data resending information table.

[0155] Step 2: Send the messages to be resent to the target queue one by one.

[0156] Step 3.1: Determine whether the resend was successful. If the resend failed, increment the sending failure count by 1 and update the sending failure count and sending failure reason information in the data resend information table.

[0157] Step 3.2: Determine if the failure threshold has been exceeded. If so, generate an early warning message to notify manual intervention.

[0158] Step 4.1: If the resending is successful and the resent data is batch data, update the number of sent records and the sending status in the data acquisition control table.

[0159] Step 4.2: Update the data resend information table to show that the sending status was successful.

[0160] Step 5: This cycle ends, pause for the set time.

[0161] The data resend component attempts to send the data to the message queue again. If the sending is successful and the data is sent in batches, the successful sending count, failed sending count, and batch sending status information in the data acquisition control table are updated. If the resend still fails after exceeding the failure threshold, the message status in the data resend information table is changed to failure, a sending failure alarm message is generated, and manual intervention is notified.

[0162] The failure threshold should be set according to the actual situation, and this application does not impose specific restrictions.

[0163] Automatic performance optimization components:

[0164] The automatic performance optimization component is used to automatically adjust the page size to optimize system performance. It sets safety boundary ranges for the key parameter PageSize, representing the amount of data per page in batch / stream transmission tasks. The initial value BasePageSize and the upper and lower limits MinPageSize and MaxPageSize are set when configuring the data acquisition configuration table. During subsequent operation, the amount of data per page in batch / stream transmission tasks is dynamically adjusted by statistically analyzing historical operation data and current load conditions. The dynamic adjustment of the amount of data per page is related to network quality, system load, and historical data transmission data. The calculation formula is shown in formula (1).

[0165] (1)

[0166] Regarding network quality, factors affecting network performance include network latency and network jitter. jitter The network bandwidth and network latency factor decrease exponentially as the latency ratio increases. The network latency factor is formed by converting the ratio of network latency to network base latency into an exponent, as shown in formula (2).

[0167] (2)

[0168] For example, when Latency = BaseLatency / 2, Nt = 1.65; when Latency = BaseLatency, Nt = 1; when Latency = BaseLatency * 1.5, Nt = 0.60.

[0169] Network adjustment factor comprehensively considers network jitter. jitter The impact of the current bandwidth usage on the Bandwidth is calculated as shown in formula (3).

[0170] (3)

[0171] Regarding system load, the main influencing factors include memory and CPU load. The ratio of available memory FreeMem to total memory TotalMem is used as the benchmark factor. Then, the influence of swap memory utilization is squared and smoothed to form the memory availability factor, the expression of which is shown in formula (4).

[0172] (4)

[0173] The system idle coefficient is used to calculate the CPU availability, and its calculation formula is shown in formula (5).

[0174] (5)

[0175] The historical success rate adjustment factor is calculated based on historical data. It uses a sliding window to calculate the ratio of successfully sent pages to the total number of sent pages in the most recent time period (10 minutes). The minimum value of this ratio is 0.37. The natural logarithm of this ratio is then added to 1 to form the historical success rate adjustment factor. This factor ranges from [0,1] and decreases rapidly logarithmically as the success rate decreases until it reaches 0.37, at which point the historical success rate adjustment factor is zero. The specific calculation formula is shown in formula (6).

[0176] (6)

[0177] Multiply the above three factors together, and then multiply by the base page size BasePageSize to generate a page value dynamically calculated by the system. To prevent deviation in page value calculation under extreme conditions, the calculation result is compared with the configured upper and lower limits MinPageSize and MaxPageSize. If it is within the safe range, it is used. If it is outside the safe range, the preset upper and lower limits are used. The specific formula is shown in formula (7).

[0178] (7)

[0179] Self-healing repair components for missions:

[0180] The task self-healing and repair component is used to automatically recover from anomalies in other components. After the task self-healing and repair process starts, it continuously monitors the status of the batch task initialization process, batch task sending process, real-time task sending process, performance automatic optimization component, and data retransmission process. When an abnormal exit is detected, these processes are restarted. For the batch task sending process, while restarting this process, the task self-healing and repair component resets the batch sending status in the data acquisition control table CM_MSG_CTL from 1 (data in transmission) to 0 (initialization) to ensure the normal status of batch data acquisition.

[0181] The data receiving component receives batch and stream-generated data from the message queue and performs subsequent data processing.

[0182] Advantages of this application:

[0183] High versatility: Implemented based on standard SQL, it controls data reading and sending via JDBC connection, and is compatible with traditional databases such as MySQL and Oracle, as well as non-relational databases such as Hive and HBase. Because standard SQL and JDBC are widely used and universal technologies, and do not depend on the logging mechanism of a specific database, it can adapt to various database types, solving the problem of poor versatility caused by reliance on database logs in existing technologies;

[0184] Supports historical batch data sending and retransmission: By adjusting the configuration table, historical batch data can be sent and retransmitted. Since the configuration table records information related to data collection and sending, by modifying the parameters and conditions in the configuration table, the data sending strategy can be flexibly controlled, including the processing of historical data, thus solving the problem that existing technologies cannot handle the retransmission of historical batch data and data with specific requirements.

[0185] Data transmission guarantee: A three-tiered guarantee system, consisting of a data transmission error self-healing mechanism, a retransmission mechanism, and an alarm mechanism, prevents data loss caused by data transmission anomalies.

[0186] Dynamic parameter adjustment: The data volume parameter for each page is dynamically adjusted to improve adaptability to different scenarios.

[0187] This application presents a dynamic paging method for transmitted data based on network quality, system load, and historical factors. Key features include a method for dynamically calculating paging based on a comprehensive consideration of network quality, system load, and historical data transmission patterns; and flexible boundary thresholds configured for different data sources. It also includes a batch-stream hybrid data transmission method and apparatus with unified configuration, featuring: methods for identifying batch and streaming data sources and identifying data changes; a data extraction component based on SQL standardization; and a triple protection method encompassing task self-healing, data retransmission, and alarm notification.

[0188] The beneficial effects of this application's embodiments are as follows: By integrating batch and real-time data transmission interfaces, a unified interface is used to complete the mixed data reception and transmission of batch and real-time data, avoiding the ambiguity caused by separate batch and stream data transmission using different interfaces. This improves the consistency, accuracy, and timeliness of mixed data, such as real-time data and batch data, during transmission. Furthermore, by adjusting the data acquisition configuration table, the transmission and retransmission of historical batch data can be achieved. Since the data acquisition configuration table records relevant information for data acquisition and transmission, modifying the parameters and conditions in the data acquisition configuration table allows for flexible control of the mixed data transmission strategy, including the processing of historical data, thus improving the flexibility of real-time and batch data transmission.

[0189] Based on the hybrid data transmission method disclosed in the above embodiments, this application also discloses a hybrid data transmission system, such as... Figure 7 As shown, the hybrid data transmission system includes:

[0190] The distinguishing unit 701 is used to distinguish the data source of the data to be processed according to the data acquisition configuration table when the data to be processed is acquired.

[0191] The first acquisition unit 702 is used to acquire batch data from the database table according to the data acquisition control table if the data to be processed comes from the database table.

[0192] The second acquisition unit 703 is used to obtain multiple data to be sent if the batch data meets the batch sending conditions;

[0193] The batch sending unit 704 is used to send multiple data to be sent to a pre-configured target queue in batches through a batch task sending process and a dynamic paging method; wherein, the dynamic paging method is a dynamic paging method that takes into account network quality, system load and historical data sending status.

[0194] The acquisition and sending unit 705 is used to acquire real-time data from the message queue in real time according to the message offset if the data to be processed comes from the message queue, and send the real-time data to the pre-configured target queue through the real-time task sending process.

[0195] Furthermore, the differentiation unit 701 is specifically used to differentiate the data source of the data to be processed according to the acquisition type in the data acquisition configuration table when the data to be processed is acquired; wherein, the acquisition type includes at least batch data and real-time data.

[0196] Furthermore, the first acquisition unit 702 is specifically used to acquire batch data from the database table according to the batch status information in the data acquisition control table if the data to be processed originates from the database table; wherein, the batch status information is determined by the batch field value, and the batch field value includes at least the date and batch identifier.

[0197] Furthermore, the second acquisition unit 703 is specifically used to determine that the batch data meets the batch sending conditions if the batch data meets the initialization of the sending batch, and to obtain multiple data to be sent;

[0198] The second acquisition unit 703 for sending batch initialization process includes:

[0199] The reading module is used to read multiple pieces of information from the data acquisition configuration table that are of batch data type.

[0200] The judgment module is used to determine whether the time interval between the current processing time and the previous processing time of multiple pieces of information is greater than the preset collection interval.

[0201] The processing module is used to determine the time interval for the next piece of information if the time interval is less than or equal to the preset collection interval.

[0202] The first query module is used to query the set of batch field names and the set of batch field values ​​that meet the collection conditions in the collection table if the time interval is greater than the preset collection interval.

[0203] The second query module is used to query the data acquisition control table by combining the set of batch field names and the set of batch field values ​​as conditions.

[0204] Add a module to the data acquisition control table if there is no matching record in the data acquisition control table. If the data acquisition control table does not have a matching record, add the acquisition table name, batch field value data and batch sending status corresponding to the combined conditions to the data acquisition control table with the initialization status.

[0205] The execution module is used to determine whether the current batch of sending tasks in the batch data has completed batch initialization if a matching record exists in the data acquisition control table, and then execute the next batch of sending tasks to complete batch initialization, until the batch initialization operation of all batch sending tasks in the batch data is completed.

[0206] Furthermore, the bulk transmission unit 704 includes:

[0207] The third query module is used to query records in the data acquisition control table whose batch sending status is initialized;

[0208] The end module is used to terminate the current loop if the query result is empty.

[0209] The conversion module is used to convert the collection table name in the collection control table and the batch field value in the corresponding batch condition into a query script if the query result is not empty and there are multiple initialized records. Here, a record represents a collection data table and the corresponding batch condition.

[0210] The execution module is used to execute the query script to retrieve the data to be sent from the query data collection control table in pages.

[0211] Multiple data messages to be sent are converted one by one into a preset standard format, and then the converted data messages are sent to the configured target queue through the batch task sending process, thus completing the process of sending multiple data messages to the pre-configured target queue in batches through the batch task sending process.

[0212] Furthermore, hybrid data transmission systems also include:

[0213] The write-start unit is used to write the unsuccessfully sent batch-initialized data into the data resend information table and start the task self-healing repair process to repair the abnormal data if an anomaly is detected during the process of sending the converted batch-initialized data to the configured target queue, and the retry sending fails after a preset number of times.

[0214] Furthermore, hybrid data transmission systems also include:

[0215] The update return unit is used to update the number of successfully sent records and the number of failed records in the data acquisition control table after each page of data is processed, and to update the batch sending status and return to query the data acquisition control table for records with the batch sending status of initialization when the last page is processed, until all records with the batch sending status of initialization in the data acquisition control table have been processed.

[0216] The beneficial effects of this application's embodiments are as follows: By integrating batch and real-time data transmission interfaces, a unified interface is used to complete the mixed data reception and transmission of batch and real-time data, avoiding the ambiguity caused by separate batch and stream data transmission using different interfaces. This improves the consistency, accuracy, and timeliness of mixed data, such as real-time data and batch data, during transmission. Furthermore, by adjusting the data acquisition configuration table, the transmission and retransmission of historical batch data can be achieved. Since the data acquisition configuration table records relevant information for data acquisition and transmission, modifying the parameters and conditions in the data acquisition configuration table allows for flexible control of the mixed data transmission strategy, including the processing of historical data, thus improving the flexibility of real-time and batch data transmission.

[0217] This application also provides a storage medium that includes stored instructions, wherein when the instructions are executed, the device containing the storage medium is controlled to perform the mixed data transmission method described above.

[0218] This application also provides an electronic device, the structural schematic diagram of which is shown below. Figure 8 As shown, it specifically includes a memory 801 and one or more instructions 802, wherein one or more instructions 802 are stored in the memory 801 and configured to be executed by one or more processors 803 to perform the above-described mixed data transmission method.

[0219] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0220] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system-type embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0221] The steps in the methods of the various embodiments of this application can be adjusted, combined, or deleted according to actual needs.

[0222] Finally, it should be noted that in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0223] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0224] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for transmitting mixed data, characterized in that, The method includes: When data to be processed is acquired, the data source of the data to be processed is distinguished according to the data acquisition configuration table; If the data to be processed originates from a database table, batch data is retrieved from the database table according to the data acquisition control table; If the batch data meets the batch sending conditions, multiple data to be sent are obtained; The multiple data items to be sent are sent in batches to a pre-configured target queue through a batch task sending process and a dynamic paging method; wherein, the dynamic paging method is a dynamic paging method that takes into account network quality, system load and historical data sending conditions. If the data to be processed originates from a message queue, real-time data is retrieved from the message queue in real time according to the message offset, and the real-time data is sent to a pre-configured target queue through a real-time task sending process.

2. The method according to claim 1, characterized in that, When data to be processed is acquired, the data source of the data to be processed is distinguished according to the data acquisition configuration table, including: When data to be processed is acquired, the data source is distinguished according to the acquisition type in the data acquisition configuration table; wherein, the acquisition type includes at least batch data and real-time data.

3. The method according to claim 1, characterized in that, If the data to be processed originates from a database table, the step of retrieving batch data from the database table based on batch status information includes: If the data to be processed originates from a database table, batch data is obtained from the database table according to the batch status information in the data acquisition control table; wherein, the batch status information is determined by the batch field value, and the batch field value includes at least the date and batch identifier.

4. The method according to claim 1, characterized in that, If the batch data meets the batch sending conditions, multiple data to be sent are obtained, including: If the batch data meets the initialization of the sending batch, it is determined that the batch data meets the batch sending conditions, and multiple data to be sent are obtained. The process of sending batch initialization includes: Read multiple records of data with batch data type from the data acquisition configuration table; For multiple pieces of information, determine whether the time interval between the current processing time and the previous processing time of the information is greater than a preset collection interval; If the time interval is less than or equal to the preset collection interval, perform time interval determination processing on the next piece of information; If the time interval is greater than the preset collection interval, query the collection table name for the set of batch field names and the set of batch field values ​​that meet the collection conditions; Use the set of batch field names and the set of batch field values ​​as combined conditions to query the data acquisition control table; If there is no matching record in the data acquisition control table, add the acquisition table name, batch field value data and batch sending status to the initialization state corresponding to the combined conditions to the data acquisition control table. If a matching record exists in the data acquisition control table, it is determined that the current batch sending task in the batch data has completed the batch initialization, and the next batch sending task is executed to complete the batch initialization, until the batch initialization operation of all batch sending tasks in the batch data is completed.

5. The method according to claim 1, characterized in that, The step of sending multiple data items to a pre-configured target queue in batches via a batch task sending process and dynamic paging includes: Query the records in the data acquisition control table whose batch sending status is initialized; If the query result is empty, end the current loop. If the query result is not empty and there are multiple initialized records, for the multiple initialized records, the collection table name in the collection control table and the batch field value in the corresponding batch condition are converted into a query script according to the records; wherein, the record represents a collection data table and the corresponding batch condition; The query script is executed to obtain the data information to be sent from the query data collection control table through dynamic pagination based on comprehensive network quality, system load, and historical data transmission status. The multiple data messages to be sent are converted one by one into a preset standard format, and the converted data messages are sent to the configured target queue through the batch task sending process, thereby completing the process of sending the multiple data messages to the pre-configured target queue in batches through the batch task sending process.

6. The method according to claim 5, characterized in that, Also includes: If an anomaly is detected during the transmission of the batch-initialized data after format conversion to the configured target queue, and the transmission fails after a preset number of retries, the unsuccessfully transmitted batch-initialized data is written into the data retransmission information table, and the task self-healing repair process is started to repair the abnormal data.

7. The method according to claim 5, characterized in that, Also includes: When each page of data is processed, the number of successfully sent records and the number of failed records in the data acquisition control table are updated. When the last page is processed, the batch sending status is updated, and the process returns to query the data acquisition control table for records with the batch sending status set to initial. This process continues until all records with the batch sending status set to initial in the data acquisition control table have been processed.

8. A hybrid data transmission system, characterized in that, The system includes: The differentiation unit is used to differentiate the data source of the data to be processed according to the data acquisition configuration table when the data to be processed is acquired. The first acquisition unit is used to acquire batch data from the database table according to the data acquisition control table if the data to be processed comes from the database table. The second acquisition unit is used to obtain multiple data to be sent if the batch data meets the batch sending conditions; The batch sending unit is used to send the multiple data to be sent to a pre-configured target queue in batches through a batch task sending process and a dynamic paging method; wherein, the dynamic paging method is a dynamic paging method that takes into account network quality, system load and historical data sending conditions; The acquisition and sending unit is used to acquire real-time data from the message queue in real time according to the message offset if the data to be processed comes from the message queue, and send the real-time data to the pre-configured target queue through the real-time task sending process.

9. A storage medium, characterized in that, The storage medium includes stored instructions, wherein, when the instructions are executed, the device containing the storage medium is controlled to perform the mixed data transmission method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes a memory, and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors as described in any one of claims 1 to 7.

Images