Data transmission method and device, computer device and storage medium

Abstract

The application relates to a data transmission method and device, computer equipment and a storage medium. The application relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a target database, a transmission task and each service data information corresponding to the transmission task; dividing each service data information into a plurality of to-be-transmitted data groups, and dividing the transmission task into a plurality of sub-transmission tasks; calling service data of a source database corresponding to the database identifier of each to-be-transmitted data group, and transmitting each service data to the target database; under the condition that a preset condition is met, adjusting the service data information in each to-be-transmitted data group, and returning to the above step until all service data are transmitted to the target database. The method can improve the efficiency of cross-source transmission in a plurality of source databases.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data transmission method, apparatus, computer device, and storage medium. Background Technology

[0002] With the rapid development of banking and financial services, financial data is also increasing dramatically. For the banking industry, financial data has become one of its most important assets. To support the diverse needs of various business scenarios, banks often store financial data in different types of databases based on the method of data retrieval (batch, online) and the data volume (large-scale, small-scale). (Database types are categorized according to the size of data stored per database unit, data storage method, and the corresponding retrieval scenario.) However, due to the complexity of banking operations, categorized storage cannot fully meet the needs of financial services in accessing financial data. Therefore, when retrieving different financial data, cross-source data scheduling (i.e., cross-database data scheduling) is required.

[0003] Traditional cross-source data transmission methods involve directly reading business data from the source databases serially, one by one, to obtain business data from multiple source databases. However, this data scheduling method is executed serially, resulting in a slow transmission rate and impacting the timeliness of the entire financial business data access link, thus leading to low efficiency in cross-source financial business data transmission. Summary of the Invention

[0004] Therefore, it is necessary to provide a data transmission method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problems.

[0005] Firstly, this application provides a data transmission method. The method includes:

[0006] Obtain the target database, the transmission task, and the service data information corresponding to the transmission task; the service data information includes the database identifier of the service data and the scheduling information of the service data;

[0007] Based on the source database corresponding to the database identifier of each business data, the business data information is divided into multiple data groups to be transmitted, and based on the business data information in each data group to be transmitted, the transmission task is divided into multiple sub-transmission tasks.

[0008] Based on the table index information of each source database, the business data of the source database corresponding to the database identifier of the business data in each data group to be transmitted is retrieved through the sub-transmission task corresponding to each data group to be transmitted, and the business data of each data group to be transmitted is transmitted to the target database.

[0009] Under the condition that the preset conditions are met, the data transmission progress information of each of the data groups to be transmitted is obtained, and the business data information in each of the data groups to be transmitted is adjusted based on the data transmission progress information. Then, the process of transmitting each of the business data to the target database is returned until all business data is transmitted to the target database.

[0010] Optionally, after obtaining the target database, the transmission task, and the various service data information corresponding to the transmission task, the method further includes:

[0011] For each business data's scheduling information containing the source database identifier, query the source database corresponding to the source database identifier of that business data.

[0012] Retrieve the table structure information of the source database and determine whether the source database has table index information;

[0013] If the source database does not have table index information, the table index information of the source database is established based on the business data contained in the source database and the table structure information.

[0014] Based on the table structure information and table index information of all source databases, the target table structure information of the target database is established.

[0015] Optionally, based on the source database corresponding to the database identifier of each business data, the business data information is divided into multiple data groups to be transmitted, including:

[0016] Based on the source database corresponding to the database identifier of each business data, the business data is classified and processed to obtain the initial data group to be transmitted corresponding to each source database.

[0017] The target initial data group to be transmitted is selected based on the average range of the number of business data information contained in all initial data groups to be transmitted.

[0018] Optionally, the method further includes:

[0019] Among all initial data groups to be transmitted, the target initial data groups to be transmitted are those whose number of business data information does not fall within the target value range.

[0020] Based on the target value range, target transmission data to be adjusted is determined in each of the target initial data to be transmitted, and data balancing processing is performed on each target initial data to be transmitted group according to the target transmission data in each of the target initial data to be transmitted to obtain the data to be transmitted group.

[0021] Optionally, based on the service data information in each group of data to be transmitted, the transmission task is divided into multiple sub-transmission tasks, including:

[0022] For each group of data to be transmitted, query the number of business data information from the same source database in the group of data to be transmitted.

[0023] In the transmission task, the unit transmission task corresponding to all the service data information contained in the data group to be transmitted is taken as the initial sub-transmission task of the data group to be transmitted.

[0024] The initial sub-transmission tasks are sorted according to the quantity of business data information from the same source database in the data group to be transmitted, from largest to smallest, and the execution order of each unit transmission task is then sorted to obtain the sub-transmission tasks.

[0025] Optionally, the step of obtaining data transmission progress information for each of the data groups to be transmitted, and adjusting the service data in each of the data groups to be transmitted based on the data transmission progress information, when preset conditions are met, includes:

[0026] Under the condition of satisfying the preset time period, query the number of untransmitted service data corresponding to each data group to be transmitted, the source database corresponding to the untransmitted service data of each data group to be transmitted, and the maximum number of service data information corresponding to each data group to be transmitted when satisfying the preset transmission load limit. For each data group to be transmitted, determine whether the data group to be transmitted has reached the preset transmission load limit.

[0027] Among the data groups to be transmitted, target data groups that reach the preset transmission load limit are selected, and the business data information in the target data groups to be transmitted is divided according to the category of the source database to obtain each sub-target data group to be transmitted.

[0028] For each target data group to be transmitted, according to the maximum number of service data information corresponding to the preset transmission load limit of the target data group to be transmitted, the first sub-target data group to be transmitted is selected from each sub-target data group to be transmitted of the target data group to be transmitted, and the first sub-target data group to be transmitted of the target data group to be transmitted is added to the data group to be transmitted that does not meet the preset transmission load limit.

[0029] Optionally, the scheduling information of the service data includes the identification information of the service data, and the method further includes:

[0030] The identification information of each transmitted service data is filled into the target table structure information of the target database to obtain the target data list;

[0031] Based on the target data list, an index relationship is established between the target data list and each of the business data to obtain the table index information of the target database.

[0032] Secondly, this application also provides a data transmission apparatus. The apparatus includes:

[0033] The acquisition module is used to acquire the target database, the transmission task, and the service data information corresponding to the transmission task; the service data information includes the database identifier of the service data and the scheduling information of the service data.

[0034] The partitioning module is used to divide each business data information into multiple data groups to be transmitted based on the source database corresponding to the database identifier of each business data, and to divide the transmission task into multiple sub-transmission tasks based on the business data information in each data group to be transmitted.

[0035] The transmission module is used to retrieve the business data of the source database corresponding to the database identifier of the business data in each data group to be transmitted, based on the table index information of each source database and through the sub-transmission task corresponding to each data group to be transmitted, and transmit each of the business data to the target database.

[0036] The adjustment module is used to obtain data transmission progress information of each of the data groups to be transmitted when preset conditions are met, adjust the business data information in each of the data groups to be transmitted based on the data transmission progress information, and return to execute the step of transmitting each of the business data to the target database until all business data has been transmitted to the target database.

[0037] Optionally, the device further includes:

[0038] The query module is used to query the source database corresponding to the source database identifier in the scheduling information of each business data.

[0039] The retrieval module is used to retrieve the table structure information of the source database and determine whether the source database has table index information.

[0040] The first establishment module is used to establish table index information of the source database based on the various business data contained in the source database and the table structure information when the source database does not have table index information.

[0041] The second module is used to establish the target table structure information of the target database based on the table structure information of all source databases and the table index information of all source databases.

[0042] Optionally, the partitioning module is specifically used for:

[0043] Based on the source database corresponding to the database identifier of each business data, the business data is classified and processed to obtain the initial data group to be transmitted corresponding to each source database.

[0044] The target initial data group to be transmitted is selected based on the average range of the number of business data information contained in all initial data groups to be transmitted.

[0045] Optionally, the device further includes:

[0046] The filtering module is used to filter out target initial data groups that do not contain business data information within a target value range from all initial data groups to be transmitted.

[0047] The processing module is used to determine the target transmission data to be adjusted in each of the target initial data to be transmitted based on the target value range, and to perform data balancing processing on each target initial data to be transmitted group according to the target transmission data in each of the target initial data to be transmitted, so as to obtain the data group to be transmitted.

[0048] Optionally, the partitioning module is specifically used for:

[0049] For each group of data to be transmitted, query the number of business data information from the same source database in the group of data to be transmitted.

[0050] In the transmission task, the unit transmission task corresponding to all the service data information contained in the data group to be transmitted is taken as the initial sub-transmission task of the data group to be transmitted.

[0051] The initial sub-transmission tasks are sorted according to the quantity of business data information from the same source database in the data group to be transmitted, from largest to smallest, and the execution order of each unit transmission task is then sorted to obtain the sub-transmission tasks.

[0052] Optionally, the adjustment module is specifically used for:

[0053] Under the condition of satisfying the preset time period, query the number of untransmitted service data corresponding to each data group to be transmitted, the source database corresponding to the untransmitted service data of each data group to be transmitted, and the maximum number of service data information corresponding to each data group to be transmitted when satisfying the preset transmission load limit. For each data group to be transmitted, determine whether the data group to be transmitted has reached the preset transmission load limit.

[0054] Among the data groups to be transmitted, target data groups that reach the preset transmission load limit are selected, and the business data information in the target data groups to be transmitted is divided according to the category of the source database to obtain each sub-target data group to be transmitted.

[0055] For each target data group to be transmitted, according to the maximum number of service data information corresponding to the preset transmission load limit of the target data group to be transmitted, the first sub-target data group to be transmitted is selected from each sub-target data group to be transmitted of the target data group to be transmitted, and the first sub-target data group to be transmitted of the target data group to be transmitted is added to the data group to be transmitted that does not meet the preset transmission load limit.

[0056] Optionally, the device further includes:

[0057] The filling module is used to fill the identification information of each transmitted service data into the target table structure information of the target database to obtain the target data list;

[0058] The third module is used to establish an index relationship between the target data list and each of the business data based on the target data list, and to obtain the table index information of the target database.

[0059] Fourthly, this application provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described in any one of the first aspects.

[0060] Fifthly, this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0061] Sixthly, this application provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0062] The aforementioned data transmission method, apparatus, computer equipment, and storage medium computer program product acquire a target database, a transmission task, and various service data information corresponding to the transmission task. The service data information includes a database identifier for the service data and scheduling information for the service data. Based on the source database corresponding to the database identifier of each service data, the service data information is divided into multiple groups of data to be transmitted. Based on the service data information in each group of data to be transmitted, the transmission task is divided into multiple sub-transmission tasks. Based on the table index information of each source database, the service data in the source database corresponding to the database identifier of the service data in each group of data to be transmitted is retrieved through the sub-transmission tasks corresponding to each group of data to be transmitted, and the service data is transmitted to the target database. Under preset conditions, data transmission progress information for each group of data to be transmitted is acquired, and the service data information in each group of data to be transmitted is adjusted based on the data transmission progress information. The process then returns to executing the step of transmitting the service data to the target database until all service data has been transmitted to the target database. By grouping and dividing the business data information corresponding to the transmission task into groups and performing data transmission simultaneously, the transmission efficiency of business data is improved. Then, by periodically acquiring the data transmission progress information of each group of data to be transmitted and adjusting the business data in each group of data to be transmitted based on the data transmission progress information, the efficiency of cross-source transmission is ensured not to decay, thereby improving the efficiency of cross-source transmission from multiple source databases. Attached Figure Description

[0063] Figure 1 This is an application environment diagram of a data transmission method in one embodiment;

[0064] Figure 2 This is a flowchart illustrating a data transmission method in one embodiment;

[0065] Figure 3 This is a flowchart illustrating the steps of adjusting the data group to be transmitted in one embodiment;

[0066] Figure 4 This is a flowchart illustrating the process of adjusting a group of data to be transmitted in one embodiment;

[0067] Figure 5 This is a flowchart illustrating a data transmission example in one embodiment;

[0068] Figure 6 This is a structural block diagram of a data transmission device in one embodiment;

[0069] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0071] The data transmission method provided in this application embodiment can be used with, for example, Figure 1 The application scenario shown includes a terminal, a source database, and a target database. The terminal retrieves business data from the source database based on each data group to be transmitted and transmits the business data from each data group to the target two-dimensional table (i.e., the target table structure information) in the target database. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, and tablets. The server can be a standalone server or a server cluster consisting of multiple servers. The terminal improves the efficiency of business data transmission by grouping and simultaneously transmitting the business data corresponding to the transmission task. Then, by periodically acquiring the data transmission progress information of each data group to be transmitted and adjusting the business data in each group based on the progress information, the efficiency of cross-source transmission is ensured to remain consistent, thus improving the efficiency of cross-source transmission from multiple source databases.

[0072] In one embodiment, such as Figure 2 As shown, a data transmission method is provided. Taking the application of this method to a terminal as an example, the method includes the following steps:

[0073] Step S201: Obtain the target database, the transmission task, and the corresponding business data information of the transmission task.

[0074] The business data information includes the database identifier of the business data and the scheduling information of the business data.

[0075] In this embodiment, the terminal responds to the user's cross-source transmission operation by acquiring the transmission command input by the user and the target database. The terminal parses the transmission command and obtains the unit transmission task, the database identifier of each service data, and the scheduling information of each service data corresponding to the transmission command. The terminal treats all unit transmission tasks as transmission tasks and the database identifier and scheduling information of each service data as service data information for each service data. The scheduling information includes the scheduling identifier and the type of the service data. Through the scheduling identifier, the service data can be retrieved from the source database corresponding to the service data.

[0076] Step S202: Based on the source database corresponding to the database identifier of each business data, divide each business data information into multiple data groups to be transmitted, and based on the business data information in each data group to be transmitted, divide the transmission task into multiple sub-transmission tasks.

[0077] In this embodiment, the terminal divides the business data information of the same source database into multiple groups based on the database identifier of each business database, resulting in multiple data groups to be transmitted. For each data group to be transmitted, the terminal takes the set of unit transmission tasks corresponding to all business data information in the data group as the sub-transmission task of that data group.

[0078] Step S203: Based on the table index information of each source database, the business data of the source database corresponding to the database identifier of the business data in each data group to be transmitted is retrieved through the sub-transmission task corresponding to each data group to be transmitted, and the business data is transmitted to the target database.

[0079] In this embodiment, the terminal retrieves the business data from the source database corresponding to each data group to be transmitted, according to the table index information of each source database and the sub-transmission tasks of the data group to be transmitted corresponding to each source database, and transmits each business data to the target database. The table index information is a two-dimensional table index in the source database. The source database based on this table index information can be stored in the source database, and the terminal retrieves the business data from this source database.

[0080] Step S204: If the preset conditions are met, obtain the data transmission progress information of each data group to be transmitted, adjust the business data information in each data group to be transmitted based on the data transmission progress information, and return to execute the step of transmitting each business data to the target database until all business data is transmitted to the target database.

[0081] In this embodiment, the terminal presets conditions for obtaining transmission progress information and determines in real time whether the current transmission progress meets the preset conditions. If the preset conditions are met, the terminal obtains the data transmission progress information of each data group to be transmitted. The preset conditions may be, but are not limited to, time intervals. If the preset time interval is met, the terminal obtains the data transmission progress information. The data transmission progress information includes the number of the data group to be transmitted, the network load of the actual underlying server where the data group to be transmitted resides, the I / O (input / output) load of the actual underlying server where the data group to be transmitted resides, the current computing resource (CPU (Central Processing Unit) utilization, memory utilization) load of the data group to be transmitted, the numbers of all data transmission subtasks currently being executed by the data group to be transmitted, and the task execution status (waiting, executing, completed) of all data transmission subtasks currently being executed by the data group to be transmitted.

[0082] Based on the data transmission progress information of each group of data to be transmitted, the terminal determines whether all service data has been transmitted to the target database. If not all service data has been transmitted to the target database, the terminal returns to step S203. If all service data has been transmitted to the target database, the terminal determines that the transmission task has been completed.

[0083] Based on the above scheme, by grouping and dividing the business data information corresponding to the transmission task and performing data transmission simultaneously, the transmission efficiency of business data is improved. Then, by periodically acquiring the data transmission progress information of each data group to be transmitted and adjusting the business data in each data group to be transmitted based on the data transmission progress information, the efficiency of cross-source transmission is ensured not to decay, and the efficiency of cross-source transmission from multiple source databases is improved.

[0084] Optionally, after obtaining the target database, the transmission task, and the corresponding business data information of the transmission task, the method further includes: querying the source database corresponding to the source database identifier in the scheduling information of each business data; retrieving the table structure information of the source database and determining whether the source database has table index information; if the source database does not have table index information, establishing the table index information of the source database based on the business data and table structure information contained in the source database; and establishing the target table structure information of the target database based on the table structure information of all source databases and the table index information of all source databases.

[0085] In this embodiment, for each service data's scheduling information containing a source database identifier, the terminal queries all source databases for the source database corresponding to that service data's source database identifier. The terminal retrieves and parses the two-dimensional table structure information of that source database. The terminal determines whether the source database contains two-dimensional table index information. If the source database contains two-dimensional table index information, the terminal retrieves and parses that index information; otherwise, the terminal establishes the two-dimensional table index information of the metadata database based on all service data in that source database and its two-dimensional table structure information. Based on the above scheme, the terminal obtains the two-dimensional table structure information and index information of all source databases. For each source database, based on its two-dimensional table structure information and index information, the terminal establishes the structural statements of the service data in the target database. Similarly, through the above scheme, the terminal obtains each structural statement of the target database, parses each structural statement, and generates the two-dimensional table structure information of the target database based on each parsed structural statement.

[0086] Based on the above scheme, by establishing a two-dimensional table structure for the target database, a data storage foundation is provided for subsequent data transmission tasks.

[0087] Optionally, based on the scheduling information of each service data, each service data information is divided into multiple data groups to be transmitted, including: classifying each service data according to the source database corresponding to the database identifier of each service data to obtain the initial data groups to be transmitted corresponding to each source database; and taking the target initial data groups to be transmitted that satisfy the average range of the number of service data information contained in all initial data groups to be transmitted as the data groups to be transmitted.

[0088] In this embodiment, the terminal determines the number of service data in the same source database based on the scheduling information of each service data, thereby obtaining the number of service data to be transmitted in each source database. For each source database, the terminal distributes the service data to be transmitted in that source database equally among the available data groups, based on the number of unloaded data groups, thus obtaining the initial data groups to be transmitted for that source database. Similarly, through the above scheme, the terminal obtains the initial data groups to be transmitted for all source databases.

[0089] Based on the above scheme, by equally dividing each initial data group to be transmitted, it is ensured that the amount of business data information in each initial data group to be transmitted meets the average number, and the transmission efficiency of each data group to be transmitted is the same, thereby improving the transmission efficiency of all data groups to be transmitted.

[0090] Optionally, the method further includes: in all initial data groups to be transmitted, filtering out target initial data groups to be transmitted whose number of included business data information does not belong to the target value range; based on the target value range, determining the target transmission data to be adjusted in each target initial data group to be transmitted, and performing data balancing processing on each target initial data group to be transmitted according to the target transmission data in each target initial data group to obtain the data group to be transmitted.

[0091] In this embodiment, the terminal calculates the average range of the number of service data information in each initial data group to be transmitted, and filters the target initial data groups to be transmitted that correspond to the average range of the number of service data information contained in all initial data groups to be transmitted. The terminal prioritizes allocating the service data information corresponding to the service data of the same source database in the service data information of the target initial data groups to be transmitted that are greater than the average range to the target initial data groups to be transmitted that are less than the average range, ensuring that the number of service data information in each target initial data group to be transmitted meets the average range. This average range is a range of values ​​with the average number as the median, fluctuating by a fixed amount above and below it. For example, if the average number is 20, the average range is [15, 25]. The terminal uses all target initial data groups to be transmitted that meet the average range of the number of service data information contained in the initial data groups to be transmitted as data groups to be transmitted, and uses all initial data groups to be transmitted that meet the average number of service data information in each initial data group as data groups to be transmitted.

[0092] Specifically, the process of evenly distributing the business data information corresponding to all pending business data based on the number of unloaded pending data groups is as follows:

[0093] The terminal periodically calculates the load of each data group awaiting transmission, including CPU (Central Processing Unit), I / O (Input / Output), memory, and network utilization; the number and details of source connections and target connections already established for the data group; and a list of pending transmission tasks (including tasks in progress and those waiting to start) and their corresponding task lists. The terminal then controls the scheduling and allocation of transmission tasks based on the load transmitted by each data group.

[0094] Based on the above scheme, by balancing the initial data groups to be transmitted for each target, a data group to be transmitted is obtained, ensuring that the amount of business data information in each data group to be transmitted is balanced, avoiding the situation where the transmission efficiency of the data group to be transmitted is too slow due to overload, thereby improving the transmission efficiency of each data group to be transmitted.

[0095] Optionally, based on the business data information in each data group to be transmitted, the transmission task is divided into multiple sub-transmission tasks, including: for each data group to be transmitted, querying the number of business data information from the same source database in the data group to be transmitted; taking the unit transmission task corresponding to all business data information contained in the data group to be transmitted as the initial sub-transmission task of the data group to be transmitted; and sorting the initial sub-transmission tasks according to the order of the number of business data information from the same source database in the data group to be transmitted from largest to smallest, and then sorting the execution order of each unit transmission task to obtain the sub-transmission task.

[0096] In this embodiment, for each data group to be transmitted, the terminal queries the quantity of business data information from the same source database within that data group. The terminal uses the unit transmission task corresponding to each business data information in that data group as the initial sub-transmission task for that data group. The terminal arranges the business data information in the data group according to the quantity, obtaining the transmission task execution order for that data group (i.e., the order from largest to smallest quantity of business data information from the same source database in the data group). The terminal then sorts each unit transmission task in the data group according to the transmission task execution order, obtaining the sub-transmission tasks for that data group. Similarly, using the above scheme, the terminal sorts the initial sub-transmission tasks for each data group to obtain the sub-transmission tasks for each data group.

[0097] Based on the above scheme, by grouping the transmission tasks, the transmission order and transmission tasks of each data group to be transmitted are ensured, thereby improving the execution efficiency of each data group to be transmitted.

[0098] Optional, such as Figure 3 As shown, under preset conditions, data transmission progress information for each data group to be transmitted is obtained, and the service data in each data group to be transmitted is adjusted based on the data transmission progress information, including:

[0099] Step S301: Under the condition of satisfying the preset time period, query the number of untransmitted service data corresponding to each data group to be transmitted, the source database corresponding to the untransmitted service data of each data group to be transmitted, and the maximum number of service data information corresponding to each data group to be transmitted when the preset transmission load limit is met, and determine whether the data group to be transmitted has reached the preset transmission load limit for each data group to be transmitted.

[0100] In this embodiment, under the preset condition of a preset time period, the terminal queries the number of untransmitted service data in each data group to be transmitted, the source database corresponding to the untransmitted service data in each data group to be transmitted, and the maximum number of service data information corresponding to each data group to be transmitted when it meets the preset transmission load limit. The obtained content includes, but is not limited to, the transmission unit number, the actual underlying server network load of the data group to be transmitted, the actual underlying server I / O (input / output) load of the data group to be transmitted, the current computing resource (CPU utilization, memory utilization) load of the data group to be transmitted, the numbers of all data transmission subtasks currently being executed by the data group to be transmitted, and the task execution status (waiting, executing, completed) of all data transmission subtasks currently being executed by the data group to be transmitted. The terminal calculates the completion progress of the corresponding data transmission task by statistically analyzing the tasks of all data transmission subtasks. The terminal sets the preset transmission load limit and, for each data group to be transmitted, determines whether the data group to be transmitted has reached the preset transmission load limit based on the number of untransmitted service data in the data group to be transmitted and the maximum number of service data information corresponding to the data group to be transmitted when it meets the preset transmission load limit. If the data group to be transmitted has not reached the preset transmission load limit, the terminal will not make any adjustments and will return to wait for the next preset time period to be met before re-executing step S102.

[0101] Step S302: In each data group to be transmitted, select the target data group to be transmitted that has reached the preset transmission load limit, and divide the business data information in the data group to be transmitted with adjustment according to the category of the source database to obtain each sub-target data group to be transmitted.

[0102] In this embodiment, when the transmitted data group reaches the preset transmission load limit, the terminal selects the data group to be transmitted that has reached the preset transmission load limit from all data groups as the target data group to be transmitted. The terminal divides the service data information in each target data group to be transmitted according to the source database corresponding to each service data information to obtain multiple sub-target data groups to be transmitted.

[0103] Step S303: For each target data group to be transmitted, according to the maximum number of service data information corresponding to the preset transmission load limit of the target data group to be transmitted, select the first sub-target data group to be transmitted from each sub-target data group to be transmitted of the target data group to be transmitted, and add the first sub-target data group to be transmitted to the data group to be transmitted that does not meet the preset transmission load limit.

[0104] In this embodiment, for each target data group to be transmitted, the terminal calculates the maximum number of service data information corresponding to the preset transmission load limit of the target data group, and the difference between the maximum number of service data information corresponding to the preset transmission load limit and the current number of service data information to be transmitted in the target data group, to obtain the target number. The terminal determines the size of the maximum number of the target data group to be transmitted and the current number of service data information to be transmitted in the target data group, and selects a data group to be transmitted that is greater than the maximum number. Among the sub-target data groups to be transmitted in the target data group, the terminal selects the sub-target data group to be transmitted with the closest number to the target number, and designates this sub-target data group as the first sub-target data group to be transmitted. Among the target data groups to be transmitted with a number less than the maximum number, the terminal selects the target target data group with the closest number to the target number of the first sub-target data group, and adjusts the first sub-target data group to be transmitted to the target target data group (that is, adds the first sub-target data group to the data group to be transmitted that does not meet the preset transmission load limit). Similarly, using the above method, the terminal adjusts and processes each target data group to be transmitted.

[0105] For example, such as Figure 4 As shown, the source databases include source database 1, source database 2, source database 3, and source database 4. The data groups to be transmitted include data group A, data group B, data group C, and data group D. When the terminal exports the business data of each metadata database to the target two-dimensional table in the target database, provided that the transmission time meets the preset time period, the terminal obtains the heartbeat information (i.e., data transmission progress information) sent by each data group to be transmitted from data group A, data group B, data group C, and data group D. Based on the sent heartbeat information, the terminal calculates and parses the load of each data group to be transmitted and determines the target data group to be transmitted. The terminal selects the first sub-target data group to be transmitted from the target data group to be transmitted and pushes the first sub-target data group to be transmitted, as well as the unit transmission tasks of each business information in the first sub-target data group to be transmitted, to data groups A, B, C, and D, thereby completing the adjustment of each data group to be transmitted.

[0106] Based on the above scheme, by periodically adjusting the business data information of each data group to be transmitted, it is ensured that all data groups to be transmitted can maintain efficient transmission, avoiding the impact on transmission efficiency due to reaching the transmission load limit, and improving the efficiency of multi-source transmission.

[0107] Optionally, the scheduling information of the business data includes the identification information of the business data. The method further includes: filling the identification information of each transmitted business data into the target table structure information of the target database to obtain the target data list; and establishing the index relationship between the target data list and each business data based on the target data list to obtain the table index information of the target database.

[0108] In this embodiment, after the terminal has completed the sub-transmission task corresponding to each of the data groups to be transmitted, it fills the identification information of each transmitted service data into the target two-dimensional table structure information (target table structure information) of the target database to obtain the target data list of the target database. Based on the target data list and the target two-dimensional table structure information of the target database, the terminal establishes an index relationship between the target data list and each service data. The terminal uses the index relationship between the target data list and all service data as the two-dimensional table index information (i.e., table index information) of the target database.

[0109] Based on the above solution, by establishing table index information in the target database, an index foundation is provided for subsequent users to retrieve business data from the target database.

[0110] This application also provides a data transmission example, as shown in Figure 5, the specific processing steps of which include:

[0111] Step S501: Obtain the target database, the transmission task, and the corresponding business data information of the transmission task.

[0112] Step S502: For each business data scheduling information source database identifier, query the source database corresponding to the source database identifier of the business data.

[0113] Step S503: Retrieve the table structure information of the source database and determine whether the source database has table index information.

[0114] Step S504: If the source database does not have table index information, establish the table index information of the source database based on the various business data and table structure information contained in the source database.

[0115] Step S505: Based on the table structure information of all source databases and the table index information of all source databases, establish the target table structure information of the target database.

[0116] Step S506: Fill the identification information of each transmitted service data into the target table structure information of the target database to obtain the target data list.

[0117] Step S507: Based on the target data list, establish the index relationship between the target data list and each business data to obtain the table index information of the target database.

[0118] Step S508: Based on the source database corresponding to the database identifier of each business data, classify and process each business data to obtain the initial data group to be transmitted corresponding to each source database.

[0119] Step S509: Select the target initial data group to be transmitted that meets the average range of the number of service data information contained in all initial data groups to be transmitted as the data group to be transmitted.

[0120] Step S510: Among all initial data groups to be transmitted, filter out target initial data groups to be transmitted whose number of business data information does not belong to the target value range.

[0121] Step S511: Based on the target value range, determine the target transmission data to be adjusted in the initial data to be transmitted for each target, and perform data balancing processing on the initial data to be transmitted for each target according to the target transmission data in the initial data to be transmitted for each target to obtain the data to be transmitted group.

[0122] Step S512: For each data group to be transmitted, query the number of business data information from the same source database in the data group to be transmitted.

[0123] Step S513: Take the unit transmission task corresponding to all the service data information contained in the data group to be transmitted in the transmission task as the initial sub-transmission task of the data group to be transmitted.

[0124] Step S514: Sort the execution order of each unit transmission task according to the quantity of business data information of the same source database in the data group to be transmitted in descending order, and obtain the sub-transmission tasks.

[0125] Step S515: Under the condition of satisfying the preset time period, query the number of untransmitted service data corresponding to each data group to be transmitted, the source database corresponding to the untransmitted service data of each data group to be transmitted, and the maximum number of service data information corresponding to each data group to be transmitted when the preset transmission load limit is met, and determine whether the data group to be transmitted has reached the preset transmission load limit for each data group to be transmitted.

[0126] Step S516: In each data group to be transmitted, select the target data group to be transmitted that reaches the preset transmission load limit, and divide the business data information in the target data group to be transmitted according to the category of the source database to obtain each sub-target data group to be transmitted.

[0127] Step S517: For each target data group to be transmitted, according to the maximum number of service data information corresponding to the preset transmission load limit of the target data group to be transmitted, select the first sub-target data group to be transmitted from each sub-target data group to be transmitted of the target data group to be transmitted, and add the first sub-target data group to be transmitted to the data group to be transmitted that does not meet the preset transmission load limit.

[0128] Step S518: Return to the step of transmitting each business data to the target database, until all business data has been transmitted to the target database.

[0129] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0130] Based on the same inventive concept, this application also provides a data transmission apparatus for implementing the data transmission method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, specific limitations in one or more data transmission apparatus embodiments provided below can be found in the limitations of the data transmission method described above, and will not be repeated here.

[0131] In one embodiment, such as Figure 6 As shown, a data transmission device is provided, including: an acquisition module 610, a division module 620, a transmission module 630, and an adjustment module 640, wherein:

[0132] The acquisition module 610 is used to acquire the target database, the transmission task, and the corresponding business data information of the transmission task; the business data information includes the database identifier of the business data and the scheduling information of the business data.

[0133] The partitioning module 620 is used to divide each business data information into multiple data groups to be transmitted based on the source database corresponding to the database identifier of each business data, and to divide the transmission task into multiple sub-transmission tasks based on the business data information in each data group to be transmitted.

[0134] The transmission module 630 is used to retrieve the business data of the source database corresponding to the database identifier of the business data in each data group to be transmitted, based on the table index information of each source database and through the sub-transmission task corresponding to each data group to be transmitted, and transmit each business data to the target database.

[0135] The adjustment module 640 is used to obtain the data transmission progress information of each data group to be transmitted when the preset conditions are met, adjust the business data information in each data group to be transmitted based on the data transmission progress information, and return to execute the step of transmitting each business data to the target database until all business data is transmitted to the target database.

[0136] Optionally, the device also includes:

[0137] The query module is used to query the source database corresponding to the source database identifier in the scheduling information of each business data.

[0138] The retrieval module is used to retrieve the table structure information of the source database and determine whether the source database has table index information.

[0139] The first module is used to create table index information for the source database based on the business data and table structure information contained in the source database when the source database does not have table index information.

[0140] The second module is used to establish the target table structure information of the target database based on the table structure information of all source databases and the table index information of all source databases.

[0141] Optionally, module 620 is used specifically for:

[0142] Based on the source database corresponding to the database identifier of each business data, the business data is classified and processed to obtain the initial data group to be transmitted corresponding to each source database.

[0143] The target initial data group to be transmitted is selected based on the average range of the number of business data information contained in all initial data groups to be transmitted.

[0144] Optionally, the device also includes:

[0145] The filtering module is used to filter out target initial data groups that do not contain business data information within a target value range from all initial data groups to be transmitted.

[0146] The processing module is used to determine the target transmission data to be adjusted in the initial data to be transmitted for each target based on the target value range, and to perform data balancing processing on the initial data to be transmitted for each target based on the target transmission data in the initial data to be transmitted for each target, so as to obtain the data to be transmitted.

[0147] Optionally, module 620 is used specifically for:

[0148] For each group of data to be transmitted, query the number of business data information from the same source database in the group of data to be transmitted;

[0149] In the transmission task, the unit transmission task corresponding to all the business data information contained in the data group to be transmitted is taken as the initial sub-transmission task of the data group to be transmitted.

[0150] The initial sub-transmission tasks are sorted according to the quantity of business data information from the same source database in the data group to be transmitted, from largest to smallest, and the execution order of each unit transmission task is obtained to obtain the sub-transmission tasks.

[0151] Optionally, adjustment module 640 is used specifically for:

[0152] Under the condition that the preset time period is met, query the number of untransmitted business data corresponding to each data group to be transmitted, the source database corresponding to the untransmitted business data of each data group to be transmitted, and the maximum number of business data information corresponding to each data group to be transmitted when the preset transmission load limit is met. For each data group to be transmitted, determine whether the data group to be transmitted has reached the preset transmission load limit.

[0153] Among the data groups to be transmitted, the target data groups that reach the preset transmission load limit are selected, and the business data information in the target data groups to be transmitted is divided according to the category of the source database to obtain each sub-target data group to be transmitted.

[0154] For each target data group to be transmitted, according to the maximum number of service data information corresponding to the preset transmission load limit of the target data group to be transmitted, the first sub-target data group to be transmitted is selected from each sub-target data group to be transmitted of the target data group to be transmitted, and the first sub-target data group to be transmitted of the target data group to be transmitted is added to the data group to be transmitted that does not meet the preset transmission load limit.

[0155] Optionally, the device also includes:

[0156] The filling module is used to fill the identification information of each transmitted service data into the target table structure information of the target database to obtain the target data list;

[0157] The third module is used to establish the index relationship between the target data list and each business data based on the target data list, and to obtain the table index information of the target database.

[0158] Each module in the aforementioned data transmission device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0159] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a data transmission method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0160] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0161] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any one of the first aspects.

[0162] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0163] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0164] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0165] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0166] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0167] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data transmission method, characterized in that, The method includes: Obtain the target database, the transmission task, and the service data information corresponding to the transmission task; the service data information includes the database identifier of the service data and the scheduling information of the service data. Based on the source database corresponding to the database identifier of each business data, the business data information is divided into multiple data groups to be transmitted, and based on the business data information in each data group to be transmitted, the transmission task is divided into multiple sub-transmission tasks. Based on the table index information of each source database, the business data of the source database corresponding to the database identifier of the business data in each data group to be transmitted is retrieved through the sub-transmission task corresponding to each data group to be transmitted, and the business data of each data group to be transmitted is transmitted to the target database. Under the condition that the preset conditions are met, the data transmission progress information of each of the data groups to be transmitted is obtained, and the business data information in each of the data groups to be transmitted is adjusted based on the data transmission progress information. Then, the process of transmitting each of the business data to the target database is returned to be executed until all business data is transmitted to the target database. The step of acquiring data transmission progress information for each of the data groups to be transmitted under preset conditions, and adjusting the service data in each of the data groups to be transmitted based on the data transmission progress information, includes: Under the condition of satisfying the preset time period, query the number of untransmitted service data corresponding to each data group to be transmitted, the source database corresponding to the untransmitted service data of each data group to be transmitted, and the maximum number of service data information corresponding to each data group to be transmitted when satisfying the preset transmission load limit. For each data group to be transmitted, determine whether the data group to be transmitted has reached the preset transmission load limit. Among the various data groups to be transmitted, target data groups that reach the preset transmission load limit are selected, and the business data information in the target data groups to be transmitted is divided according to the category of the source database to obtain each sub-target data group to be transmitted. For each target data group to be transmitted, according to the maximum number of service data information corresponding to the preset transmission load limit of the target data group to be transmitted, the first sub-target data group to be transmitted is selected from each sub-target data group to be transmitted of the target data group to be transmitted, and the first sub-target data group to be transmitted of the target data group to be transmitted is added to the data group to be transmitted that does not meet the preset transmission load limit.

2. The method according to claim 1, characterized in that, After obtaining the target database, the transmission task, and the various service data information corresponding to the transmission task, the method further includes: For each business data's scheduling information containing the source database identifier, query the source database corresponding to the source database identifier of that business data. Retrieve the table structure information of the source database and determine whether the source database has table index information; If the source database does not have table index information, the table index information of the source database is established based on the business data contained in the source database and the table structure information. Based on the table structure information and table index information of all source databases, the target table structure information of the target database is established.

3. The method according to claim 1, characterized in that, The source database corresponding to the database identifier of each business data item divides each business data item into multiple data groups to be transmitted, including: Based on the source database corresponding to the database identifier of each business data, the business data is classified and processed to obtain the initial data group to be transmitted corresponding to each source database. The target initial data group to be transmitted is selected based on the average range of the number of business data information contained in all initial data groups to be transmitted.

4. The method according to claim 3, characterized in that, The method further includes: Among all initial data groups to be transmitted, the target initial data groups to be transmitted are those whose number of business data information does not fall within the target value range. Based on the target value range, target transmission data to be adjusted is determined in each of the target initial data to be transmitted, and data balancing processing is performed on each target initial data to be transmitted group according to the target transmission data in each of the target initial data to be transmitted to obtain the data to be transmitted group.

5. The method according to claim 1, characterized in that, Based on the service data information in each group of data to be transmitted, the transmission task is divided into multiple sub-transmission tasks, including: For each group of data to be transmitted, query the number of business data information from the same source database in the group of data to be transmitted; In the transmission task, the unit transmission task corresponding to all the service data information contained in the data group to be transmitted is taken as the initial sub-transmission task of the data group to be transmitted. The initial sub-transmission tasks are sorted according to the quantity of business data information from the same source database in the data group to be transmitted, from largest to smallest, and the execution order of each unit transmission task is then obtained to obtain the sub-transmission tasks.

6. The method according to claim 2, characterized in that, The scheduling information of the service data includes the identification information of the service data, and the method further includes: The identification information of each transmitted service data is filled into the target table structure information of the target database to obtain the target data list; Based on the target data list, an index relationship is established between the target data list and each of the business data to obtain the table index information of the target database.

7. A data transmission device, characterized in that, The device includes: The acquisition module is used to acquire the target database, the transmission task, and the service data information corresponding to the transmission task; the service data information includes the database identifier of the service data and the scheduling information of the service data. The partitioning module is used to divide each business data information into multiple data groups to be transmitted based on the source database corresponding to the database identifier of each business data, and to divide the transmission task into multiple sub-transmission tasks based on the business data information in each data group to be transmitted. The transmission module is used to retrieve the business data of the source database corresponding to the database identifier of the business data in each data group to be transmitted, based on the table index information of each source database and through the sub-transmission task corresponding to each data group to be transmitted, and transmit each of the business data to the target database. The adjustment module is used to obtain data transmission progress information of each of the data groups to be transmitted when the preset conditions are met, adjust the business data information in each of the data groups to be transmitted based on the data transmission progress information, and return to execute the step of transmitting each of the business data to the target database until all business data is transmitted to the target database. The adjustment module is specifically used to query the number of untransmitted service data corresponding to each data group to be transmitted, the source database corresponding to the untransmitted service data of each data group to be transmitted, and the maximum number of service data information corresponding to each data group to be transmitted when the preset transmission load limit is met, and to determine whether the data group to be transmitted has reached the preset transmission load limit for each data group to be transmitted. Among the various data groups to be transmitted, target data groups that reach the preset transmission load limit are selected, and the business data information in the target data groups to be transmitted is divided according to the category of the source database to obtain each sub-target data group to be transmitted. For each target data group to be transmitted, according to the maximum number of service data information corresponding to the preset transmission load limit of the target data group to be transmitted, the first sub-target data group to be transmitted is selected from each sub-target data group to be transmitted of the target data group to be transmitted, and the first sub-target data group to be transmitted of the target data group to be transmitted is added to the data group to be transmitted that does not meet the preset transmission load limit.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images