Data transmission method, device, electronic device, storage medium and program product

By storing massive data into the intermediate data component and requesting downloads from the database when needed, the database IO peak and performance degradation caused by massive data writing after processing by the big data platform, the database IO balance and performance improvement are achieved.

CN112597234BActive Publication Date: 2024-11-26LAKALA PAYMENT CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202011597431.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-29
Publication Date
2024-11-26
Estimated Expiration
2040-12-29

AI Technical Summary

Technical Problem

After the existing technology is processed on the big data platform, when massive data is directly written to the database, it will lead to peak database IO, performance degradation, and may even cause database paralysis.

Method used

Store the massive data to be transmitted to the data intermediate component instead of sending it directly to the target database. The target database requests download data when needed according to its own data transmission situation, and uses the massive data transfer storage mechanism.

Benefits of technology

It effectively avoids the peaks and performance of database IO, ensures the IO balance of database, and improves database performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112597234B_ABST
    Figure CN112597234B_ABST
Patent Text Reader

Abstract

The disclosed embodiments disclose a data transmission method, device, electronic device, storage medium and program product, the method comprising: obtaining target mass data; generating a target data file based on the target mass data; sending the target data file to a data intermediate component, so that the target database can request the data intermediate component to download the target data file when needed. This technical solution can not only effectively avoid database IO peaks, database IO imbalance, database performance degradation, poor user experience, and even database operation paralysis caused by the writing of mass data or sub-massive data by means of a mass data transfer storage mechanism, but also ensure database IO balance and improve database performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of data processing technology, and in particular to a data transmission method, device, electronic device, storage medium, and program product. Background Art

[0002] With the development of science and technology and Internet technology, more and more massive data need to be processed and transmitted. In the prior art, after the big data platform completes processing or calculation, massive data or sub-massive data is generated, and the generated massive data or sub-massive data is pushed to the corresponding database, the massive data or sub-massive data is usually directly written into the corresponding database. However, since the big data platform does not understand the data flow of the database receiving the data, the writing of massive data or sub-massive data often causes the database to generate IO (input and output) peaks, affects IO balance, reduces database performance, and deteriorates user experience. In severe cases, it may even cause the database to paralyze. Summary of the invention

[0003] The embodiments of the present disclosure provide a data transmission method, device, electronic device, storage medium, and program product.

[0004] In a first aspect, a data transmission method is provided in an embodiment of the present disclosure.

[0005] Specifically, the data transmission method includes:

[0006] Obtain massive target data;

[0007] Generate a target data file based on the target mass data;

[0008] The target data file is sent to the data intermediate component, so that the target database can request the data intermediate component to download the target data file when needed.

[0009] In combination with the first aspect, in a first implementation manner of the first aspect of the embodiment of the present disclosure, the number of the target data files is one or more.

[0010] In combination with the first aspect and the first implementation of the first aspect, in a second implementation of the first aspect of the embodiment of the present disclosure, when there are multiple target data files, generating the target data file based on the target massive data includes:

[0011] Obtaining a preset data length, and dividing the target mass data into a plurality of target data files according to the preset data length;

[0012] Generate a target data index file, wherein the target data index file includes identification information of the plurality of target data files and generation order information of the plurality of target data files;

[0013] The sending of the target data file to the data intermediate component is implemented as follows:

[0014] The plurality of target data files and the target data index file are sent to a data intermediate component.

[0015] In combination with the first aspect, the first implementation manner of the first aspect and the second implementation manner of the first aspect, in a third implementation manner of the first aspect of the present disclosure, the data intermediate component is an intermediate file server or a message middleware.

[0016] In combination with the first aspect, the first implementation of the first aspect, the second implementation of the first aspect, and the third implementation of the first aspect, the fourth implementation of the first aspect of the present disclosure further includes:

[0017] In response to confirming that the target data file is transmitted completely, a target data file reception prompt message is sent to the target database, wherein the target data file reception prompt message carries the data intermediate component address storing the target data file and the target data file attribute information.

[0018] In a second aspect, a data transmission method is provided in an embodiment of the present disclosure.

[0019] Specifically, the data transmission method includes:

[0020] Receiving a target data file reception prompt message sent by a target platform, wherein the target data file reception prompt message carries a data intermediate component address storing the target data file and target data file attribute information;

[0021] In response to confirming that the data transmission condition is met, acquiring the data intermediate component address storing the target data file and the target data file attribute information from the target data file reception prompt message;

[0022] Request the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information.

[0023] In combination with the second aspect, in the first implementation method of the second aspect of the embodiment of the present disclosure, the data transmission condition includes one or more of the following conditions: the available time for data transmission is greater than a preset time threshold, the available bandwidth for data transmission is greater than a preset bandwidth threshold, and the number of available resources for data transmission is greater than a preset number threshold.

[0024] In combination with the second aspect and the first implementation manner of the second aspect, in the second implementation manner of the second aspect of the embodiment of the present disclosure, the target data file is one or more.

[0025] In combination with the second aspect, the first implementation of the second aspect, and the second implementation of the second aspect, in a third implementation of the second aspect of the present disclosure, when there are multiple target data files, requesting the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information includes:

[0026] Determine a target data intermediate component according to the data intermediate component address;

[0027] According to the target data file attribute information, a request is made to the target data intermediate component to download a target data index file and a plurality of target data files corresponding to the target data file attribute information.

[0028] In combination with the second aspect, the first implementation of the second aspect, the second implementation of the second aspect, and the third implementation of the second aspect, the fourth implementation of the second aspect of the present disclosure further includes:

[0029] The multiple target data files are combined into target mass data according to the target data index file.

[0030] In a third aspect, a data transmission method is provided in an embodiment of the present disclosure.

[0031] Specifically, the data transmission method includes:

[0032] The target platform acquires target mass data, generates a target data file based on the target mass data, sends the target data file to the data intermediate component, and in response to confirming that the target data file is transmitted, sends a target data file reception prompt message to the target database, wherein the target data file reception prompt message carries the data intermediate component address storing the target data file and the target data file attribute information;

[0033] The target database receives a target data file reception prompt message sent by the target platform, and in response to confirming that the data transmission conditions are met, obtains the data intermediate component address and target data file attribute information storing the target data file from the target data file reception prompt message, and requests the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information.

[0034] In combination with the third aspect, in a first implementation manner of the third aspect of the embodiment of the present disclosure, the number of the target data files is one or more.

[0035] In combination with the third aspect and the first implementation of the third aspect, in the second implementation of the third aspect of the embodiment of the present disclosure, when there are multiple target data files,

[0036] The generating a target data file based on the target mass data includes:

[0037] Obtaining a preset data length, and dividing the target mass data into a plurality of target data files according to the preset data length; generating a target data index file, wherein the target data index file includes identification information of the plurality of target data files and generation order information of the plurality of target data files;

[0038] The sending of the target data file to the data intermediate component is implemented as follows:

[0039] Sending the plurality of target data files and the target data index file to a data intermediate component;

[0040] The step of requesting the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information includes:

[0041] Determine a target data intermediate component according to the data intermediate component address;

[0042] According to the target data file attribute information, a request is made to the target data intermediate component to download a target data index file and a plurality of target data files corresponding to the target data file attribute information.

[0043] In combination with the third aspect, the first implementation manner of the third aspect and the second implementation manner of the third aspect, in a third implementation manner of the third aspect of the present disclosure, requesting the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information further includes:

[0044] The multiple target data files are combined into target mass data according to the target data index file.

[0045] In combination with the third aspect, the first implementation of the third aspect, the second implementation of the third aspect and the third implementation of the third aspect, in a fourth implementation of the third aspect of the present disclosure, the data intermediate component is an intermediate file server or a message middleware.

[0046] In a fourth aspect, a data transmission device is provided in an embodiment of the present disclosure.

[0047] Specifically, the data transmission device includes:

[0048] A first acquisition module is configured to acquire target mass data;

[0049] A generating module, configured to generate a target data file based on the target mass data;

[0050] The first sending module is configured to send the target data file to the data intermediate component, so that the target database can request the data intermediate component to download the target data file when needed.

[0051] In combination with the fourth aspect, in a first implementation manner of the fourth aspect of an embodiment of the present disclosure, the number of the target data files is one or more.

[0052] In combination with the fourth aspect and the first implementation manner of the fourth aspect, in the second implementation manner of the fourth aspect of the embodiment of the present disclosure, when there are multiple target data files, the generating module is configured as follows:

[0053] Obtaining a preset data length, and dividing the target mass data into a plurality of target data files according to the preset data length;

[0054] Generate a target data index file, wherein the target data index file includes identification information of the plurality of target data files and generation order information of the plurality of target data files;

[0055] The first sending module is configured as follows:

[0056] The plurality of target data files and the target data index file are sent to a data intermediate component.

[0057] In combination with the fourth aspect, the first implementation manner of the fourth aspect and the second implementation manner of the fourth aspect, in a third implementation manner of the fourth aspect of the present disclosure, the data intermediate component is an intermediate file server or a message middleware.

[0058] In combination with the fourth aspect, the first implementation of the fourth aspect, the second implementation of the fourth aspect, and the third implementation of the fourth aspect, the present disclosure, in a fourth implementation of the fourth aspect, further includes:

[0059] The second sending module is configured to send a target data file reception prompt message to the target database in response to confirming that the target data file transmission is completed, wherein the target data file reception prompt message carries the data intermediate component address storing the target data file and the target data file attribute information.

[0060] In a fifth aspect, a data transmission device is provided in an embodiment of the present disclosure.

[0061] Specifically, the data transmission device includes:

[0062] A receiving module is configured to receive a target data file reception prompt message sent by a target platform, wherein the target data file reception prompt message carries a data intermediate component address storing the target data file and target data file attribute information;

[0063] A second acquisition module is configured to, in response to confirming that the data transmission condition is met, acquire the data intermediate component address storing the target data file and the target data file attribute information from the target data file reception prompt message;

[0064] The download module is configured to request the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information.

[0065] In combination with the fifth aspect, in the first implementation method of the fifth aspect of the embodiment of the present disclosure, the data transmission condition includes one or more of the following conditions: the available time for data transmission is greater than a preset time threshold, the available bandwidth for data transmission is greater than a preset bandwidth threshold, and the number of available resources for data transmission is greater than a preset number threshold.

[0066] In combination with the fifth aspect and the first implementation manner of the fifth aspect, in the second implementation manner of the fifth aspect of the embodiment of the present disclosure, the target data file is one or more.

[0067] In combination with the fifth aspect, the first implementation manner of the fifth aspect and the second implementation manner of the fifth aspect, in a third implementation manner of the fifth aspect of the present disclosure, when there are multiple target data files, the download module is configured as follows:

[0068] Determine a target data intermediate component according to the data intermediate component address;

[0069] According to the target data file attribute information, a request is made to the target data intermediate component to download a target data index file and a plurality of target data files corresponding to the target data file attribute information.

[0070] In combination with the fifth aspect, the first implementation manner of the fifth aspect, the second implementation manner of the fifth aspect, and the third implementation manner of the fifth aspect, the present disclosure, in a fourth implementation manner of the fifth aspect, further includes:

[0071] The combining module is configured to combine the multiple target data files into target mass data according to the target data index file.

[0072] In a sixth aspect, a data transmission device is provided in an embodiment of the present disclosure.

[0073] Specifically, the data transmission device includes:

[0074] The target platform is configured to obtain target mass data, generate a target data file based on the target mass data, send the target data file to the data intermediate component, and in response to confirming that the target data file is transmitted, send a target data file reception prompt message to the target database, wherein the target data file reception prompt message carries the data intermediate component address storing the target data file and the target data file attribute information;

[0075] The target database is configured to receive a target data file reception prompt message sent by the target platform, and in response to confirming that the data transmission conditions are met, obtain the data intermediate component address and target data file attribute information storing the target data file from the target data file reception prompt message, and request the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information.

[0076] In combination with the sixth aspect, in a first implementation manner of the sixth aspect of an embodiment of the present disclosure, the number of the target data files is one or more.

[0077] In combination with the sixth aspect and the first implementation of the sixth aspect, in the second implementation of the sixth aspect of the embodiment of the present disclosure, when there are multiple target data files,

[0078] The part of the target platform that generates the target data file based on the target massive data is configured as follows:

[0079] Obtaining a preset data length, and dividing the target mass data into a plurality of target data files according to the preset data length; generating a target data index file, wherein the target data index file includes identification information of the plurality of target data files and generation order information of the plurality of target data files;

[0080] The part of the target platform that sends the target data file to the data intermediate component is configured as follows:

[0081] Sending the plurality of target data files and the target data index file to a data intermediate component;

[0082] The part of the target database requesting the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information is configured as follows:

[0083] Determine a target data intermediate component according to the data intermediate component address;

[0084] According to the target data file attribute information, a request is made to the target data intermediate component to download a target data index file and a plurality of target data files corresponding to the target data file attribute information.

[0085] In combination with the sixth aspect, the first implementation manner of the sixth aspect, and the second implementation manner of the sixth aspect, in a third implementation manner of the sixth aspect of the present disclosure, the part of the target database requesting the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information is further configured as follows:

[0086] The multiple target data files are combined into target mass data according to the target data index file.

[0087] In combination with the sixth aspect, the first implementation of the sixth aspect, the second implementation of the sixth aspect and the third implementation of the sixth aspect, in a fourth implementation of the sixth aspect of the present disclosure, the data intermediate component is an intermediate file server or a message middleware.

[0088] In a seventh aspect, an embodiment of the present disclosure provides an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer instructions that support a data transmission device to perform the above-mentioned data transmission method, and the processor is configured to execute the computer instructions stored in the memory. The data transmission device may also include a communication interface for the data transmission device to communicate with other devices or a communication network.

[0089] In an eighth aspect, an embodiment of the present disclosure provides a computer-readable storage medium for storing computer instructions used by a data transmission device, which includes computer instructions involved in the data transmission device for executing the above-mentioned data transmission method.

[0090] In a ninth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned data transmission method when executed by a processor.

[0091] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0092] The above technical solution stores the massive data to be transmitted in the data intermediate component instead of sending it directly to the target database, so that the target database can request the data intermediate component to download the massive data when needed according to its own data transmission situation. This technical solution can not only effectively avoid database IO peaks, database IO imbalance, database performance degradation, poor user experience, and even database operation paralysis caused by the writing of massive data or sub-massive data by means of the massive data transfer storage mechanism, but also ensure the IO balance of the database and improve the performance of the database.

[0093] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Other features, purposes and advantages of the embodiments of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0095] Figure 1 A flow chart showing a data transmission method according to an embodiment of the present disclosure;

[0096] Figure 2 A flowchart showing a data transmission method according to another embodiment of the present disclosure is shown;

[0097] Figure 3 A flowchart showing a data transmission method according to another embodiment of the present disclosure is shown;

[0098] Figure 4 An overall flow chart of a data transmission method according to an embodiment of the present disclosure is shown;

[0099] Figure 5 A structural block diagram of a data transmission device according to an embodiment of the present disclosure is shown;

[0100] Figure 6 A structural block diagram of a data transmission device according to another embodiment of the present disclosure is shown;

[0101] Figure 7 A structural block diagram of a data transmission system according to another embodiment of the present disclosure is shown;

[0102] Figure 8 It is a structural diagram of a computer system suitable for implementing a data transmission method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0103] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0104] In the embodiments of the present disclosure, it should be understood that terms such as "including" or "having" are intended to indicate the existence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in this specification, and are not intended to exclude the possibility of one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof existing or being added.

[0105] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0106] The technical solution provided by the disclosed embodiment stores the massive data to be transmitted in the data intermediate component instead of sending it directly to the target database, so that the target database can request the data intermediate component to download the massive data when needed according to its own data transmission situation. This technical solution can not only effectively avoid database IO peaks, database IO imbalance, database performance degradation, poor user experience, and even database operation paralysis caused by the writing of massive data or sub-massive data by means of the massive data transfer storage mechanism, but also ensure the IO balance of the database and improve the performance of the database.

[0107] Figure 1 A flow chart of a data transmission method according to an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the data transmission method includes the following steps S101-S103:

[0108] In step S101, target mass data is obtained;

[0109] In step S102, a target data file is generated based on the target mass data;

[0110] In step S103, the target data file is sent to the data intermediate component, so that the target database can request the data intermediate component to download the target data file when needed.

[0111] As mentioned above, with the development of science and technology and Internet technology, more and more massive data need to be processed and transmitted. In the prior art, when the big data platform completes processing or calculation, generates massive data or sub-massive data, and pushes the generated massive data or sub-massive data to the corresponding database, the massive data or sub-massive data is usually written directly to the corresponding database. However, since the big data platform does not understand the data flow of the database receiving the data, the writing of massive data or sub-massive data often causes the database to generate IO (input and output) peaks, affects IO balance, reduces database performance, and deteriorates user experience. In severe cases, it may even cause the database to paralyze.

[0112] In view of the above problems, in this embodiment, a data transmission method is proposed, which stores the massive data to be transmitted in the data intermediate component instead of sending it directly to the target database, so that the target database can request the data intermediate component to download the massive data when needed according to its own data transmission situation. This technical solution can not only effectively avoid database IO peaks, database IO imbalance, database performance degradation, poor user experience, and even database operation paralysis caused by the writing of massive data or sub-massive data by means of the massive data transfer storage mechanism, but also ensure the IO balance of the database and improve the performance of the database.

[0113] In one embodiment of the present disclosure, the data transmission method may be applicable to devices such as a big data platform that can perform data transmission.

[0114] In one embodiment of the present disclosure, the target massive data refers to massive data to be transmitted, for example, massive data or sub-massive data generated by the big data platform and need to be transmitted to the target database.

[0115] In one embodiment of the present disclosure, the target data file refers to a data file that is generated based on the target mass data to be transmitted and is more convenient to transmit and store.

[0116] In one embodiment of the present disclosure, the data intermediate component refers to a component that has the ability to store massive or sub-massive data and can realize the transfer of massive or sub-massive data, such as an FTP file server, a message middleware such as kafka and pulsar that supports a message consumption mechanism, etc. When the data intermediate component is an FTP file server, the big data platform exports the target data file to the FTP file server, and the target database can subsequently download the target data file with the help of an FTP download instruction; when the data intermediate component is a message middleware such as kafka and pulsar that supports a message consumption mechanism, the big data platform can write the target data file into the message middleware, and the target database can subsequently obtain the target data file with the help of a message consumption request.

[0117] In the above implementation, after the big data platform obtains the target massive data to be transmitted through processing or calculation, it generates a target data file that is convenient for transmission and storage based on the target massive data, and then sends the target data file to the data intermediate component for transit storage. The target database that subsequently receives the target massive data can request the data intermediate component to download the target data file when it needs it. The above transit storage mechanism can not only effectively avoid database IO peaks, database IO imbalance, database performance degradation, poor user experience, and even database operation paralysis caused by the writing of massive data or sub-massive data, but also ensure the IO balance of the database and improve the performance of the database.

[0118] In one embodiment of the present disclosure, in order to facilitate the transmission and storage of the target data file, the target data file may be set to be multiple, that is, the target data file may be one or more.

[0119] When there are multiple target data files, step S102, i.e., the step of generating the target data file based on the target mass data, may include the following steps:

[0120] Obtaining a preset data length, and dividing the target mass data into a plurality of target data files according to the preset data length;

[0121] A target data index file is generated, wherein the target data index file includes identification information of the plurality of target data files and generation order information of the plurality of target data files.

[0122] In order to facilitate the transmission and storage of the target data files, in this embodiment, the target mass data is divided into multiple target data files and then the multiple target data files are transmitted and stored separately. Specifically, first obtain a preset data length, wherein the preset data length can be set according to the needs of the actual application and the characteristics of the target mass data, and the present disclosure does not specifically limit it; then divide the target mass data into multiple target data files according to the preset data length, wherein each target data file is provided with corresponding identification information to distinguish the target data files; then generate a target data index file according to the identification information of the target data file and the generation time or sequence information of different target data files, and subsequently, after the target database downloads the target data index file and the corresponding target data file, it can combine the target data files in sequence according to the target data file identification information and the target data file generation sequence information in the target data index file to restore the target mass data.

[0123] In this embodiment, the step S103, i.e. the step of sending the target data file to the data intermediate component, can be implemented as follows:

[0124] The plurality of target data files and the target data index file are sent to a data intermediate component.

[0125] In this way, the subsequent target database can request the data intermediate component to download the multiple target data files and the target data index file when needed, and combine the target data files to obtain the target mass data.

[0126] In one embodiment of the present disclosure, the method further comprises the following steps:

[0127] In response to confirming that the target data file is transmitted completely, a target data file reception prompt message is sent to the target database, wherein the target data file reception prompt message carries the data intermediate component address storing the target data file and the target data file attribute information.

[0128] In order to enable the target database to understand the transmission status of the target data file, download the target data file in time, and obtain the target massive data, in this embodiment, when the big data platform confirms that the transmission of the target data file is completed, it sends a target data file reception prompt message to the target database to prompt the target database to request to download the target data file. The target data file reception prompt message may carry the data intermediate component address storing the target data file and the target data file attribute information. The target database can reach the data intermediate component storing the target data file according to the data intermediate component address, and download the target data file according to the target data file attribute information, wherein the target data file attribute information may be, for example, the identification information of the target data file and other information.

[0129] Among them, when there are multiple target data files, the big data platform can send a target data file reception prompt message to the target database after confirming that one or more target data files have been transferred. After the target database has downloaded all target data files, the target data files are combined according to the target data index file to form the target massive data, thereby realizing time-sharing transmission of the target data files, further ensuring the IO balance of the database, and improving the performance of the database.

[0130] Figure 2 A flow chart of a data transmission method according to another embodiment of the present disclosure is shown. Figure 2 As shown, the data transmission method includes the following steps S201-S203:

[0131] In step S201, a target data file reception prompt message sent by a target platform is received, wherein the target data file reception prompt message carries the data intermediate component address storing the target data file and the target data file attribute information;

[0132] In step S202, in response to confirming that the data transmission condition is met, the data intermediate component address storing the target data file and the target data file attribute information are obtained from the target data file receiving prompt message;

[0133] In step S203, a request is made to the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information.

[0134] As mentioned above, with the development of science and technology and Internet technology, more and more massive data need to be processed and transmitted. In the prior art, when the big data platform completes processing or calculation, generates massive data or sub-massive data, and pushes the generated massive data or sub-massive data to the corresponding database, the massive data or sub-massive data is usually written directly to the corresponding database. However, since the big data platform does not understand the data flow of the database receiving the data, the writing of massive data or sub-massive data often causes the database to generate IO (input and output) peaks, affects IO balance, reduces database performance, and deteriorates user experience. In severe cases, it may even cause the database to paralyze.

[0135] In view of the above problems, in this embodiment, a data transmission method is proposed, which stores the massive data to be transmitted in the data intermediate component instead of sending it directly to the target database, so that the target database can request the data intermediate component to download the massive data when needed according to its own data transmission situation. This technical solution can not only effectively avoid database IO peaks, database IO imbalance, database performance degradation, poor user experience, and even database operation paralysis caused by the writing of massive data or sub-massive data by means of the massive data transfer storage mechanism, but also ensure the IO balance of the database and improve the performance of the database.

[0136] In one embodiment of the present disclosure, the data transmission method may be applicable to a device such as a target database that receives the transmission data.

[0137] In one embodiment of the present disclosure, the target platform refers to a platform that generates target massive data to be transmitted, such as a big data platform and the like.

[0138] In one embodiment of the present disclosure, the data transmission condition refers to the condition that the target database needs to meet to execute the data download request, wherein the data transmission condition may include, for example, one or more of the following conditions: the available time for data transmission is greater than a preset time threshold, wherein the available time for data transmission may be, for example, an IO idle time; the available bandwidth for data transmission is greater than a preset bandwidth threshold; the number of available resources for data transmission is greater than a preset number threshold, wherein the available resources for data transmission may be, for example, a data transmission device, a data transmission component, and the like.

[0139] In the above implementation, after the target database receives the target data file reception prompt message sent by the target platform and confirms that the preset data transmission conditions are met, the data intermediate component address and target data file attribute information storing the target data file can be obtained from the target data file reception prompt message; then, the data intermediate component storing the target data file is determined according to the data intermediate component address, and a request is made to the data intermediate component to download the target data file corresponding to the target data file attribute information according to the target data file attribute information.

[0140] In one embodiment of the present disclosure, the target data file is one or more.

[0141] In this embodiment, when there are multiple target data files, step S203, i.e., the step of requesting the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information, may include the following steps:

[0142] Determine a target data intermediate component according to the data intermediate component address;

[0143] According to the target data file attribute information, a request is made to the target data intermediate component to download a target data index file and a plurality of target data files corresponding to the target data file attribute information.

[0144] In this embodiment, the method may further include the following steps:

[0145] The multiple target data files are combined into target mass data according to the target data index file.

[0146] As mentioned above, in order to facilitate the transmission and storage of the target data files, the big data platform can divide the target mass data into multiple target data files for transmission and storage respectively, and simultaneously generate a target data index file including identification information and generation order information of multiple target data files. Therefore, in this embodiment, when there are multiple target data files, in order to download all the target data files, the target database first determines the target data intermediate component according to the data intermediate component address, and then requests the target data intermediate component to download the target data index file and multiple target data files corresponding to the target data file attribute information according to the target data file attribute information, and then combines the downloaded multiple target data files into the target mass data according to the identification information of the multiple target data files included in the target data index file and the generation order information of the multiple target data files.

[0147] In one embodiment of the present disclosure, if the target data file or the target massive data recovered from the target data file is inconsistent with the characteristics such as the data format that the target database can receive, the target data file or the target massive data recovered from the target data file can be written into the target database after a preset parsing operation is performed on it.

[0148] Figure 2 The technical terms and technical features involved in the embodiments shown and related Figure 1 The technical terms and technical features mentioned in the embodiments shown and related are the same or similar. Figure 2 The explanation and description of the technical terms and technical features involved in the embodiments shown and related can refer to the above Figure 1 The explanations of the illustrated and related embodiments will not be repeated here.

[0149] Figure 3 A flow chart of a data transmission method according to another embodiment of the present disclosure is shown as follows: Figure 3 As shown, the data transmission method includes the following steps S301-S302:

[0150] In step S301, the target platform acquires target mass data, generates a target data file based on the target mass data, sends the target data file to the data intermediate component, and in response to confirming that the target data file transmission is completed, sends a target data file reception prompt message to the target database, wherein the target data file reception prompt message carries the data intermediate component address storing the target data file and the target data file attribute information;

[0151] In step S302, the target database receives a target data file reception prompt message sent by the target platform, and in response to confirming that the data transmission conditions are met, obtains the data intermediate component address and target data file attribute information storing the target data file from the target data file reception prompt message, and requests the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information.

[0152] As mentioned above, with the development of science and technology and Internet technology, more and more massive data need to be processed and transmitted. In the prior art, when the big data platform completes processing or calculation, generates massive data or sub-massive data, and pushes the generated massive data or sub-massive data to the corresponding database, the massive data or sub-massive data is usually written directly to the corresponding database. However, since the big data platform does not understand the data flow of the database receiving the data, the writing of massive data or sub-massive data often causes the database to generate IO (input and output) peaks, affects IO balance, reduces database performance, and deteriorates user experience. In severe cases, it may even cause the database to paralyze.

[0153] In view of the above problems, in this embodiment, a data transmission method is proposed, which stores the massive data to be transmitted in the data intermediate component instead of sending it directly to the target database, so that the target database can request the data intermediate component to download the massive data when needed according to its own data transmission situation. This technical solution can not only effectively avoid database IO peaks, database IO imbalance, database performance degradation, poor user experience, and even database operation paralysis caused by the writing of massive data or sub-massive data by means of the massive data transfer storage mechanism, but also ensure the IO balance of the database and improve the performance of the database.

[0154] In one embodiment of the present disclosure, the data transmission method may be applicable to a data transmission system including a big data platform capable of executing data transmission and a target database for receiving the transmitted data.

[0155] In one embodiment of the present disclosure, the target data file is one or more.

[0156] In one embodiment of the present disclosure, when there are multiple target data files,

[0157] The generating a target data file based on the target mass data includes:

[0158] Obtaining a preset data length, and dividing the target mass data into a plurality of target data files according to the preset data length; generating a target data index file, wherein the target data index file includes identification information of the plurality of target data files and generation order information of the plurality of target data files;

[0159] The sending of the target data file to the data intermediate component is implemented as follows:

[0160] Sending the plurality of target data files and the target data index file to a data intermediate component;

[0161] The step of requesting the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information includes:

[0162] Determine a target data intermediate component according to the data intermediate component address;

[0163] According to the target data file attribute information, a request is made to the target data intermediate component to download a target data index file and a plurality of target data files corresponding to the target data file attribute information.

[0164] In one embodiment of the present disclosure, the requesting the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information further includes:

[0165] The multiple target data files are combined into target mass data according to the target data index file.

[0166] In one embodiment of the present disclosure, the data intermediate component is an intermediate file server or a message middleware.

[0167] Figure 3 The technical terms and technical features involved in the embodiments shown and related Figure 1-2 The technical terms and technical features mentioned in the embodiments shown and related are the same or similar. Figure 3 The explanation and description of the technical terms and technical features involved in the embodiments shown and related can refer to the above Figure 1-2 The explanations of the illustrated and related embodiments will not be repeated here.

[0168] Figure 4 The overall flow chart of the data transmission method according to an embodiment of the present disclosure is shown as follows: Figure 4 As shown, the target platform obtains target massive data, generates a target data file based on the target massive data, exports the target data file to an FTP file server, or writes it to a message middleware such as kafka, pulsar, etc. that supports a message consumption mechanism. When the target database confirms that the data transmission conditions are met, it downloads the target data file from the FTP file server with the help of an FTP download instruction, or obtains the target data file with the help of a message consumption request. When necessary, the target data file or the target massive data recovered from the target data file can be parsed and then written into the target database.

[0169] The following are embodiments of the apparatus of the present disclosure, which can be used to execute embodiments of the method of the present disclosure.

[0170] Figure 5FIG. 1 shows a structural block diagram of a data transmission device according to an embodiment of the present disclosure. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 5 As shown, the data transmission device includes:

[0171] A first acquisition module 501 is configured to acquire target mass data;

[0172] A generating module 502 is configured to generate a target data file based on the target mass data;

[0173] The first sending module 503 is configured to send the target data file to the data intermediate component, so that the target database can request the data intermediate component to download the target data file when needed.

[0174] As mentioned above, with the development of science and technology and Internet technology, more and more massive data need to be processed and transmitted. In the prior art, when the big data platform completes processing or calculation, generates massive data or sub-massive data, and pushes the generated massive data or sub-massive data to the corresponding database, the massive data or sub-massive data is usually written directly to the corresponding database. However, since the big data platform does not understand the data flow of the database receiving the data, the writing of massive data or sub-massive data often causes the database to generate IO (input and output) peaks, affects IO balance, reduces database performance, and deteriorates user experience. In severe cases, it may even cause the database to paralyze.

[0175] Considering the above problems, in this embodiment, a data transmission device is proposed, which stores the massive data to be transmitted in the data intermediate component instead of sending it directly to the target database, so that the target database can request the data intermediate component to download the massive data when needed according to its own data transmission situation. This technical solution can not only effectively avoid database IO peaks, database IO imbalance, database performance degradation, poor user experience, and even database operation paralysis caused by the writing of massive data or sub-massive data by means of the massive data transfer storage mechanism, but also ensure the IO balance of the database and improve the performance of the database.

[0176] In one embodiment of the present disclosure, the data transmission device can be implemented as a device such as a big data platform that can perform data transmission.

[0177] In one embodiment of the present disclosure, the target massive data refers to massive data to be transmitted, for example, massive data or sub-massive data generated by the big data platform and need to be transmitted to the target database.

[0178] In one embodiment of the present disclosure, the target data file refers to a data file that is generated based on the target mass data to be transmitted and is more convenient to transmit and store.

[0179] In one embodiment of the present disclosure, the data intermediate component refers to a component that has the ability to store massive or sub-massive data and can realize the transfer of massive or sub-massive data, such as an FTP file server, a message middleware such as kafka and pulsar that supports a message consumption mechanism, etc. When the data intermediate component is an FTP file server, the big data platform exports the target data file to the FTP file server, and the target database can subsequently download the target data file with the help of an FTP download instruction; when the data intermediate component is a message middleware such as kafka and pulsar that supports a message consumption mechanism, the big data platform can write the target data file into the message middleware, and the target database can subsequently obtain the target data file with the help of a message consumption request.

[0180] In the above implementation, after the big data platform obtains the target massive data to be transmitted through processing or calculation, it generates a target data file that is convenient for transmission and storage based on the target massive data, and then sends the target data file to the data intermediate component for transit storage. The target database that subsequently receives the target massive data can request the data intermediate component to download the target data file when it needs it. The above transit storage mechanism can not only effectively avoid database IO peaks, database IO imbalance, database performance degradation, poor user experience, and even database operation paralysis caused by the writing of massive data or sub-massive data, but also ensure the IO balance of the database and improve the performance of the database.

[0181] In one embodiment of the present disclosure, in order to facilitate the transmission and storage of the target data file, the target data file may be set to be multiple, that is, the target data file may be one or more.

[0182] When there are multiple target data files, the generating module 502 may be configured as follows:

[0183] Obtaining a preset data length, and dividing the target mass data into a plurality of target data files according to the preset data length;

[0184] A target data index file is generated, wherein the target data index file includes identification information of the plurality of target data files and generation order information of the plurality of target data files.

[0185] In order to facilitate the transmission and storage of the target data files, in this embodiment, the target mass data is divided into multiple target data files and then the multiple target data files are transmitted and stored separately. Specifically, first obtain a preset data length, wherein the preset data length can be set according to the needs of the actual application and the characteristics of the target mass data, and the present disclosure does not specifically limit it; then divide the target mass data into multiple target data files according to the preset data length, wherein each target data file is provided with corresponding identification information to distinguish the target data files; then generate a target data index file according to the identification information of the target data file and the generation time or sequence information of different target data files, and subsequently, after the target database downloads the target data index file and the corresponding target data file, it can combine the target data files in sequence according to the target data file identification information and the target data file generation sequence information in the target data index file to restore the target mass data.

[0186] In this implementation, the first sending module 503 may be configured as follows:

[0187] The plurality of target data files and the target data index file are sent to a data intermediate component.

[0188] In this way, the subsequent target database can request the data intermediate component to download the multiple target data files and the target data index file when needed, and combine the target data files to obtain the target mass data.

[0189] In one embodiment of the present disclosure, the device further comprises:

[0190] The second sending module is configured to send a target data file reception prompt message to the target database in response to confirming that the target data file transmission is completed, wherein the target data file reception prompt message carries the data intermediate component address storing the target data file and the target data file attribute information.

[0191] In order to enable the target database to understand the transmission status of the target data file, download the target data file in time, and obtain the target massive data, in this embodiment, when the big data platform confirms that the transmission of the target data file is completed, it sends a target data file reception prompt message to the target database to prompt the target database to request to download the target data file. The target data file reception prompt message may carry the data intermediate component address storing the target data file and the target data file attribute information. The target database can reach the data intermediate component storing the target data file according to the data intermediate component address, and download the target data file according to the target data file attribute information, wherein the target data file attribute information may be, for example, the identification information of the target data file and other information.

[0192] Among them, when there are multiple target data files, the big data platform can send a target data file reception prompt message to the target database after confirming that one or more target data files have been transferred. After the target database has downloaded all target data files, the target data files are combined according to the target data index file to form the target massive data, thereby realizing time-sharing transmission of the target data files, further ensuring the IO balance of the database, and improving the performance of the database.

[0193] Figure 6 FIG. 2 shows a structural block diagram of a data transmission device according to another embodiment of the present disclosure, which can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 6 As shown, the data transmission device includes:

[0194] The receiving module 601 is configured to receive a target data file reception prompt message sent by a target platform, wherein the target data file reception prompt message carries a data intermediate component address storing the target data file and target data file attribute information;

[0195] The second acquisition module 602 is configured to, in response to confirming that the data transmission condition is met, acquire the data intermediate component address storing the target data file and the target data file attribute information from the target data file receiving prompt message;

[0196] The download module 603 is configured to request the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information.

[0197] As mentioned above, with the development of science and technology and Internet technology, more and more massive data need to be processed and transmitted. In the prior art, when the big data platform completes processing or calculation, generates massive data or sub-massive data, and pushes the generated massive data or sub-massive data to the corresponding database, the massive data or sub-massive data is usually written directly to the corresponding database. However, since the big data platform does not understand the data flow of the database receiving the data, the writing of massive data or sub-massive data often causes the database to generate IO (input and output) peaks, affects IO balance, reduces database performance, and deteriorates user experience. In severe cases, it may even cause the database to paralyze.

[0198] Considering the above problems, in this embodiment, a data transmission device is proposed, which stores the massive data to be transmitted in the data intermediate component instead of sending it directly to the target database, so that the target database can request the data intermediate component to download the massive data when needed according to its own data transmission situation. This technical solution can not only effectively avoid database IO peaks, database IO imbalance, database performance degradation, poor user experience, and even database operation paralysis caused by the writing of massive data or sub-massive data by means of the massive data transfer storage mechanism, but also ensure the IO balance of the database and improve the performance of the database.

[0199] In one embodiment of the present disclosure, the data transmission device can be implemented as a device such as a target database that receives the transmission data.

[0200] In one embodiment of the present disclosure, the target platform refers to a platform that generates target massive data to be transmitted, such as a big data platform and the like.

[0201] In one embodiment of the present disclosure, the data transmission condition refers to the condition that the target database needs to meet to execute the data download request, wherein the data transmission condition may include, for example, one or more of the following conditions: the available time for data transmission is greater than a preset time threshold, wherein the available time for data transmission may be, for example, an IO idle time; the available bandwidth for data transmission is greater than a preset bandwidth threshold; the number of available resources for data transmission is greater than a preset number threshold, wherein the available resources for data transmission may be, for example, a data transmission device, a data transmission component, and the like.

[0202] In the above implementation, after the target database receives the target data file reception prompt message sent by the target platform and confirms that the preset data transmission conditions are met, the data intermediate component address and target data file attribute information storing the target data file can be obtained from the target data file reception prompt message; then, the data intermediate component storing the target data file is determined according to the data intermediate component address, and a request is made to the data intermediate component to download the target data file corresponding to the target data file attribute information according to the target data file attribute information.

[0203] In one embodiment of the present disclosure, the target data file is one or more.

[0204] In this implementation, when there are multiple target data files, the download module 603 may be configured as follows:

[0205] Determine a target data intermediate component according to the data intermediate component address;

[0206] According to the target data file attribute information, a request is made to the target data intermediate component to download a target data index file and a plurality of target data files corresponding to the target data file attribute information.

[0207] In this embodiment, the device may further include:

[0208] The combining module is configured to combine the multiple target data files into target mass data according to the target data index file.

[0209] As mentioned above, in order to facilitate the transmission and storage of the target data files, the big data platform can divide the target mass data into multiple target data files for transmission and storage respectively, and simultaneously generate a target data index file including identification information and generation order information of multiple target data files. Therefore, in this embodiment, when there are multiple target data files, in order to download all the target data files, the target database first determines the target data intermediate component according to the data intermediate component address, and then requests the target data intermediate component to download the target data index file and multiple target data files corresponding to the target data file attribute information according to the target data file attribute information, and then combines the downloaded multiple target data files into the target mass data according to the identification information of the multiple target data files included in the target data index file and the generation order information of the multiple target data files.

[0210] In one embodiment of the present disclosure, if the target data file or the target massive data recovered from the target data file is inconsistent with the characteristics such as the data format that the target database can receive, the target data file or the target massive data recovered from the target data file can be written into the target database after a preset parsing operation is performed on it.

[0211] Figure 6 The technical terms and technical features involved in the embodiments shown and related Figure 5 The technical terms and technical features mentioned in the embodiments shown and related are the same or similar. Figure 6 The explanation and description of the technical terms and technical features involved in the embodiments shown and related can refer to the above Figure 5 The explanations of the illustrated and related embodiments will not be repeated here.

[0212] Figure 7 FIG. 2 shows a structural block diagram of a data transmission device according to another embodiment of the present disclosure, which can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 7 As shown, the data transmission device includes:

[0213] The target platform 701 is configured to obtain target mass data, generate a target data file based on the target mass data, send the target data file to the data intermediate component, and in response to confirming that the target data file is transmitted, send a target data file reception prompt message to the target database, wherein the target data file reception prompt message carries the data intermediate component address storing the target data file and the target data file attribute information;

[0214] The target database 702 is configured to receive a target data file reception prompt message sent by the target platform, and in response to confirming that the data transmission conditions are met, obtain the data intermediate component address and target data file attribute information storing the target data file from the target data file reception prompt message, and request the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information.

[0215] As mentioned above, with the development of science and technology and Internet technology, more and more massive data need to be processed and transmitted. In the prior art, when the big data platform completes processing or calculation, generates massive data or sub-massive data, and pushes the generated massive data or sub-massive data to the corresponding database, the massive data or sub-massive data is usually written directly to the corresponding database. However, since the big data platform does not understand the data flow of the database receiving the data, the writing of massive data or sub-massive data often causes the database to generate IO (input and output) peaks, affects IO balance, reduces database performance, and deteriorates user experience. In severe cases, it may even cause the database to paralyze.

[0216] Considering the above problems, in this embodiment, a data transmission device is proposed, which stores the massive data to be transmitted in the data intermediate component instead of sending it directly to the target database, so that the target database can request the data intermediate component to download the massive data when needed according to its own data transmission situation. This technical solution can not only effectively avoid database IO peaks, database IO imbalance, database performance degradation, poor user experience, and even database operation paralysis caused by the writing of massive data or sub-massive data by means of the massive data transfer storage mechanism, but also ensure the IO balance of the database and improve the performance of the database.

[0217] In one embodiment of the present disclosure, the data transmission device can be implemented as a data transmission system including a big data platform capable of executing data transmission and a target database for receiving the transmitted data.

[0218] In one embodiment of the present disclosure, the target data file is one or more.

[0219] In one embodiment of the present disclosure, when there are multiple target data files,

[0220] The part of the target platform that generates the target data file based on the target mass data can be configured as follows:

[0221] Obtaining a preset data length, and dividing the target mass data into a plurality of target data files according to the preset data length; generating a target data index file, wherein the target data index file includes identification information of the plurality of target data files and generation order information of the plurality of target data files;

[0222] The part of the target platform that sends the target data file to the data intermediate component can be configured as follows:

[0223] Sending the plurality of target data files and the target data index file to a data intermediate component;

[0224] The part of the target database requesting the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information can be configured as follows:

[0225] Determine a target data intermediate component according to the data intermediate component address;

[0226] According to the target data file attribute information, a request is made to the target data intermediate component to download a target data index file and a plurality of target data files corresponding to the target data file attribute information.

[0227] In one embodiment of the present disclosure, the part of the target database requesting the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information is further configured as follows:

[0228] The multiple target data files are combined into target mass data according to the target data index file.

[0229] In one embodiment of the present disclosure, the data intermediate component is an intermediate file server or a message middleware.

[0230] Figure 7 The technical terms and technical features involved in the embodiments shown and related Figure 5-6 The technical terms and technical features mentioned in the embodiments shown and related are the same or similar. Figure 7 The explanation and description of the technical terms and technical features involved in the embodiments shown and related can refer to the above Figure 5-6 The explanations of the illustrated and related embodiments will not be repeated here.

[0231] The present disclosure also discloses an electronic device, which includes a memory and a processor; wherein:

[0232] The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement any of the above method steps.

[0233] Figure 8 It is a structural diagram of a computer system suitable for implementing a data transmission method according to an embodiment of the present disclosure.

[0234] like Figure 8As shown, the computer system 800 includes a processing unit 801, which can perform various processes in the above-mentioned embodiments according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the computer system 800 are also stored. The processing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0235] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. The drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that the computer program read therefrom is installed into the storage section 808 as needed. Among them, the processing unit 801 can be implemented as a processing unit such as a CPU, a GPU, a TPU, an FPGA, an NPU, etc.

[0236] In particular, according to an embodiment of the present disclosure, the method described above can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a readable medium thereof, and the computer program includes a program code for executing the data transmission method. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 809, and / or installed from a removable medium 811.

[0237] The embodiment of the present disclosure also discloses a computer program product, which includes a computer program / instruction, and when the computer program / instruction is executed by a processor, any of the above method steps is implemented.

[0238] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the road map or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a different order from the order marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0239] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0240] As another aspect, the embodiments of the present disclosure further provide a computer-readable storage medium, which may be a computer-readable storage medium included in the device described in the above-mentioned embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the methods described in the embodiments of the present disclosure.

[0241] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A data transmission method, comprising: Acquire target massive data, wherein the target massive data refers to massive data or sub-massive data that needs to be transmitted to a target database; Generating a target data file based on the target mass data, wherein the target data file refers to a data file that is more convenient to transmit and store and is generated based on the target mass data to be transmitted; Sending the target data file to the data intermediate component, so that the target database can request the data intermediate component to download the target data file when needed, wherein the data intermediate component refers to a component with massive or sub-massive data storage capability and capable of transferring massive or sub-massive data; Wherein, the target data file is one or more. When the target data file is multiple, generating the target data file based on the target mass data includes: Obtaining a preset data length, and dividing the target mass data into a plurality of target data files according to the preset data length; Generate a target data index file, wherein the target data index file includes identification information of the plurality of target data files and generation order information of the plurality of target data files; The sending of the target data file to the data intermediate component is implemented as follows: Sending the plurality of target data files and the target data index file to a data intermediate component; Among them, the data intermediate component is an intermediate file server or message middleware with massive or sub-massive data storage capabilities and capable of realizing massive or sub-massive data transfer. When the data intermediate component is an intermediate file server, the target data file is exported to the intermediate file server, and subsequently the target database downloads the target data file with the help of a download instruction. When the data intermediate component is a message middleware, the target data file is written into the message middleware, and subsequently the target database obtains the target data file with the help of a message consumption request.

2. The method according to claim 1, further comprising: In response to confirming that the target data file is transmitted completely, a target data file reception prompt message is sent to the target database, wherein the target data file reception prompt message carries the data intermediate component address storing the target data file and the target data file attribute information.

3. A data transmission method, comprising: Receive a target data file reception prompt message sent by the target platform, wherein the target data file reception prompt message carries the data intermediate component address storing the target data file and the target data file attribute information, wherein the target data file refers to a data file that is more convenient to transmit and store, generated based on the target mass data to be transmitted, and the data intermediate component refers to a component with mass or sub-massive data storage capacity and capable of realizing mass or sub-massive data transfer; In response to confirming that the data transmission condition is met, acquiring the data intermediate component address storing the target data file and the target data file attribute information from the target data file reception prompt message; Requesting the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information; Wherein, the target data file is one or more. When the target data file is multiple, requesting the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information includes: Determine a target data intermediate component according to the data intermediate component address; According to the target data file attribute information, a request is made to the target data intermediate component to download a target data index file and a plurality of target data files corresponding to the target data file attribute information.

4. According to the method according to claim 3, the data transmission conditions include one or more of the following conditions: the available time for data transmission is greater than a preset time threshold, the available bandwidth for data transmission is greater than a preset bandwidth threshold, and the number of available resources for data transmission is greater than a preset number threshold.

5. The method according to claim 3, further comprising: The multiple target data files are combined into target mass data according to the target data index file.

6. A data transmission method, comprising: The target platform acquires target mass data, generates a target data file based on the target mass data, sends the target data file to the data intermediate component, and in response to confirming that the target data file is transmitted, sends a target data file reception prompt message to the target database, wherein the target mass data refers to mass data or sub-massive data that needs to be transmitted to the target database, the target data file refers to a data file that is more convenient to transmit and store that is generated based on the target mass data to be transmitted, the data intermediate component refers to an intermediate file server or message middleware that has mass or sub-massive data storage capabilities and can realize mass or sub-massive data transfer, and the target data file reception prompt message carries the data intermediate component address storing the target data file and target data file attribute information; The target database receives a target data file receiving prompt message sent by the target platform, obtains the data intermediate component address storing the target data file and the target data file attribute information from the target data file receiving prompt message in response to confirming that the data transmission condition is met, and requests the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information; Wherein, the target data file is one or more. When the target data file is multiple, The generating a target data file based on the target mass data comprises: Obtaining a preset data length, and dividing the target mass data into a plurality of target data files according to the preset data length; generating a target data index file, wherein the target data index file includes identification information of the plurality of target data files and generation order information of the plurality of target data files; The sending of the target data file to the data intermediate component is implemented as follows: Sending the plurality of target data files and the target data index file to a data intermediate component; The step of requesting the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information includes: Determine a target data intermediate component according to the data intermediate component address; According to the target data file attribute information, a request is made to the target data intermediate component to download a target data index file and a plurality of target data files corresponding to the target data file attribute information.

7. The method according to claim 6, wherein the requesting the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information further comprises: The multiple target data files are combined into target mass data according to the target data index file.

8. A data transmission device, comprising: A first acquisition module is configured to acquire target massive data, wherein the target massive data refers to massive data or sub-massive data that needs to be transmitted to a target database; A generating module, configured to generate a target data file based on the target mass data, wherein the target data file refers to a data file that is generated based on the target mass data to be transmitted and is more convenient to transmit and store; a first sending module, configured to send the target data file to a data intermediate component, so that the target database can request the data intermediate component to download the target data file when needed, wherein the data intermediate component refers to a component with a massive or sub-massive data storage capability and capable of transferring massive or sub-massive data; Wherein, the target data file is one or more. When the target data file is multiple, the generating module is configured as follows: Obtaining a preset data length, and dividing the target mass data into a plurality of target data files according to the preset data length; Generate a target data index file, wherein the target data index file includes identification information of the plurality of target data files and generation order information of the plurality of target data files; The first sending module is configured as follows: Sending the plurality of target data files and the target data index file to a data intermediate component; Among them, the data intermediate component is an intermediate file server or message middleware with massive or sub-massive data storage capabilities and capable of realizing massive or sub-massive data transfer. When the data intermediate component is an intermediate file server, the target data file is exported to the intermediate file server, and subsequently the target database downloads the target data file with the help of a download instruction. When the data intermediate component is a message middleware, the target data file is written into the message middleware, and subsequently the target database obtains the target data file with the help of a message consumption request.

9. The apparatus according to claim 8, further comprising: The second sending module is configured to send a target data file reception prompt message to the target database in response to confirming that the target data file transmission is completed, wherein the target data file reception prompt message carries the data intermediate component address storing the target data file and the target data file attribute information.

10. A data transmission device, comprising: A receiving module is configured to receive a target data file reception prompt message sent by a target platform, wherein the target data file reception prompt message carries a data intermediate component address storing the target data file and target data file attribute information; A second acquisition module is configured to, in response to confirming that the data transmission condition is met, acquire the data intermediate component address storing the target data file and the target data file attribute information from the target data file receiving prompt message, wherein the target data file refers to a data file that is more convenient to transmit and store generated based on the target mass data to be transmitted, and the data intermediate component refers to a component with mass or sub-massive data storage capacity and capable of realizing mass or sub-massive data transfer; A download module, configured to request the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information; Wherein, the target data file is one or more. When the target data file is multiple, the download module is configured as follows: Determine a target data intermediate component according to the data intermediate component address; According to the target data file attribute information, a request is made to the target data intermediate component to download a target data index file and a plurality of target data files corresponding to the target data file attribute information.

11. According to the device of claim 10, the data transmission condition includes one or more of the following conditions: the available time for data transmission is greater than a preset time threshold, the available bandwidth for data transmission is greater than a preset bandwidth threshold, and the number of available resources for data transmission is greater than a preset number threshold.

12. The apparatus according to claim 10, further comprising: The combining module is configured to combine the multiple target data files into target mass data according to the target data index file.

13. A data transmission device, comprising: The target platform is configured to obtain target mass data, generate a target data file based on the target mass data, send the target data file to the data intermediate component, and send a target data file reception prompt message to the target database in response to confirming that the target data file transmission is complete, wherein the target mass data refers to mass data or sub-massive data that needs to be transmitted to the target database, the target data file refers to a data file that is more convenient to transmit and store that is generated based on the target mass data to be transmitted, the data intermediate component refers to an intermediate file server or message middleware that has mass or sub-massive data storage capabilities and can realize mass or sub-massive data transfer, and the target data file reception prompt message carries the data intermediate component address storing the target data file and target data file attribute information; The target database is configured to receive a target data file reception prompt message sent by the target platform, and in response to confirming that the data transmission condition is met, obtain the data intermediate component address and target data file attribute information storing the target data file from the target data file reception prompt message, and request the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information; Wherein, the target data file is one or more. When the target data file is multiple, The part of the target platform that generates the target data file based on the target massive data is configured as follows: Obtaining a preset data length, and dividing the target mass data into a plurality of target data files according to the preset data length; generating a target data index file, wherein the target data index file includes identification information of the plurality of target data files and generation order information of the plurality of target data files; The part of the target platform that sends the target data file to the data intermediate component is configured as follows: Sending the plurality of target data files and the target data index file to a data intermediate component; The part of the target database requesting the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information is configured as follows: Determine a target data intermediate component according to the data intermediate component address; According to the target data file attribute information, a request is made to the target data intermediate component to download a target data index file and a plurality of target data files corresponding to the target data file attribute information.

14. The apparatus according to claim 13, wherein the part of the target database requesting the data intermediate component to download the target data file according to the data intermediate component address and the target data file attribute information is further configured as follows: The multiple target data files are combined into target mass data according to the target data index file.

15. An electronic device comprising a memory and a processor; wherein: The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the steps of any one of claims 1-7.

16. A computer-readable storage medium having computer instructions stored thereon, wherein: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

17. A computer program product, comprising a computer program / instruction, which implements the steps of the method according to any one of claims 1 to 7 when executed by a processor.

Citation Information

Patent Citations

  • Video-on-demand method and device

    CN105338378A

  • Real-time transmission processing method, server and system of mass data

    CN105978968A

  • Information processing method and device

    CN106850242A

  • Media file acquisition method and device and storage medium

    CN110309342A

  • Data migration method based on message-oriented middleware and related equipment

    CN110334075A