Efficient data transmission method and device

Abstract

The invention relates to the technical field of data transmission, and particularly discloses an efficient data transmission method and device. The efficient data transmission method comprises the following steps: (1) generating a to-be-stored data object, and putting the to-be-stored data object into a to-be-stored data object list; (2) checking a to-be-stored data object list in real time, when a to-be-stored data object exists in the to-be-stored data object list, establishing a cache file according to the time sequence, naming the cache file according to a preset format, and writing the to-be-stored data object into the corresponding cache file; (3) obtaining a local cache file name list, and removing the files of which the names do not conform to the preset format from the local cache file name list; and (4) judging whether the number of the files in the local cache file name list is greater than the maximum number of the historical cache files or not. According to the invention, the problems of system blockage and data loss can be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to an efficient data transmission method and device. Background Art

[0002] In most business systems, the generated data needs to be persisted to data storage terminals such as databases or off-site media. The data is generated continuously and no blockage is allowed. After a large amount of data is generated, it is directly written to the database or off-site media and other data storage terminals through network IO. Once a storage bottleneck occurs, the system resource usage will be too high. If the data storage terminal cannot be connected, the business system data will accumulate and then be lost, which will directly affect the normal business operation of the system, especially the real-time data monitoring system. Summary of the invention

[0003] The present invention provides an efficient data transmission method and device, which can solve the problems of system blocking and data loss.

[0004] To solve the above problems, the present invention adopts the following technical solutions:

[0005] According to a first aspect of the present invention, an embodiment of the present invention provides an efficient data transmission method, comprising the following steps:

[0006] (1) Generate a data object to be stored and put the data object to be stored into a list of data objects to be stored;

[0007] (2) checking the list of data objects to be stored in real time, and when there is a data object to be stored in the list of data objects to be stored, creating a cache file according to the time series, naming the cache file according to a preset format, and writing the data object to be stored into the corresponding cache file;

[0008] (3) obtaining a local cache file name list, and removing files whose names do not conform to the preset format from the local cache file name list;

[0009] (4) determining whether the number of files in the local cache file name list is greater than the maximum number of historical cache files;

[0010] (5) If the number of files is greater than the maximum number of historical cache files, remove the files one by one in chronological order, starting from the oldest file, until the number of files is less than or equal to the maximum number of historical cache files; if the number of files is less than or equal to the maximum number of historical cache files, proceed directly to the subsequent steps;

[0011] (6) traverse and process all files in the local cache file name list to determine whether the time difference between the system time and the start time of file processing is greater than the cache file list processing timeout;

[0012] (7) If the time difference between the system time and the start time of the file processing is greater than the cache file list processing timeout, return to step (3); if the time difference between the system time and the start time of the file processing is less than or equal to the cache file list processing timeout, directly proceed to the subsequent steps;

[0013] (8) Open the cache file in the local cache file name list, read the data object to be stored, and transmit the data object to be stored to the data storage end according to the business logic;

[0014] (9) Determine whether the data object to be stored is successfully transmitted;

[0015] (10) If the data object to be stored is transmitted successfully, continue to process the next cache file until all the data objects to be stored are transmitted successfully; if the data object to be stored is not transmitted successfully, return to step (3).

[0016] In some embodiments, in step (2), if there is no data object to be stored in the list of data objects to be stored, the process continues to wait in step (2).

[0017] In some embodiments, the step of establishing a cache file based on a time series and naming the cache file according to a preset format specifically includes: defining the moment when the list of data objects to be stored is obtained as tmBegin, traversing and processing the list of data objects to be stored, defining the subscript of the current data object to be stored as i, i starts from 0, defining the current time of the i-th data object to be stored as tmNow, and assigning tmNow to tmBegin plus i milliseconds; the name of the cache file includes the current time tmNow of the i-th data object to be stored and a preset identification field.

[0018] In some embodiments, step (3) specifically includes: periodically obtaining a list of local cache file names, filtering out files whose names contain the identification field, and adding a removal field to the names of files whose name formats do not conform to the preset format, and removing files whose names contain the removal field from the list of local cache file names.

[0019] In some embodiments, when there is no cache file containing the identification field in the local cache file name list, the period for obtaining the local cache file name list is extended.

[0020] In some embodiments, between naming the cache file according to a preset format and writing the data object to be stored into the corresponding cache file, the following steps are also included: determining whether the created cache file already exists locally; if the created cache file already exists locally, deleting the existing local file.

[0021] In some embodiments, the step of writing the data object to be stored into the corresponding cache file specifically includes: writing the data object to be stored into the cache file according to a defined cache file format, wherein the cache file format includes 41-character identification information, a 4-character version number and business data arranged in sequence.

[0022] In some embodiments, the following steps are also included between step (5) and step (6): traversing all cache files in the local cache file name list, comparing the current time tmNow of the i-th data object to be stored in the file name with the current system time, and when the absolute value of the difference between the two times is less than the time threshold, removing the file from the local cache file name list; when the number of files in the local cache file name list is zero, entering step (3).

[0023] In some embodiments, after step (10), the following step is also included: periodically obtaining a local cache file name list, and deleting files whose names contain the removal field from the local cache file name list.

[0024] According to a second aspect of the present invention, an embodiment of the present invention provides an efficient data transmission device, comprising:

[0025] A data generation module, used for generating a data object to be stored, and putting the data object to be stored into a list of data objects to be stored;

[0026] The cache file module is used to check the list of data objects to be stored in real time. When there is a data object to be stored in the list of data objects to be stored, a cache file is created according to the time series, and the cache file is named according to a preset format, and the data object to be stored is written into the corresponding cache file;

[0027] A file sorting module is used to obtain a local cache file name list and remove files whose names do not conform to the preset format from the local cache file name list;

[0028] The first judgment module is used to judge whether the number of files in the local cache file name list is greater than the maximum number of historical cache files;

[0029] A file removal module, for removing files one by one in time sequence starting from the earliest file when the first judgment module judges that the number of files is greater than the maximum number of historical cache files, until the number of files is less than or equal to the maximum number of historical cache files;

[0030] The second judgment module is used to traverse and process all the files in the local cache file name list when the first judgment module determines that the number of files is less than or equal to the maximum number of historical cache files, and determine whether the time difference between the system time and the start time of file processing is greater than the cache file list processing timeout time;

[0031] The file arrangement module is also used to obtain a local cache file name list when the time difference between the system time and the start processing time of the file is greater than the cache file list processing timeout time, and remove files whose names do not conform to the preset format from the local cache file name list;

[0032] The file opening module is used to open the cache file in the local cache file name list when the time difference between the system time and the start processing time of the file is less than or equal to the cache file list processing timeout time, read the data object to be stored, and transmit the data object to be stored to the data storage end according to the business logic;

[0033] A third judgment module is used to judge whether the data object to be stored is successfully transmitted;

[0034] A continuing processing module is used to continue processing the next cache file when the data object to be stored is successfully transmitted, until all the data objects to be stored are successfully transmitted;

[0035] The file arrangement module is also used to obtain a local cache file name list when the data object to be stored is not successfully transmitted, and remove files whose names do not conform to the preset format from the local cache file name list.

[0036] The present invention has at least the following beneficial effects: the present invention uses multi-threaded asynchronous processing to first write the data object to be stored into the cache file, thereby reducing the business system blocking, reducing the business system memory usage, improving the business system response speed, and removing files that do not conform to the preset format to avoid the local non-cached files from being transmitted together, thereby reducing the occupation of the system channel; when the number of local cache files is greater than the maximum number of historical cache files, the local cache files are removed, and the number of cache files is controlled to prevent the number of cache files from increasing indefinitely and affecting the overall operation of the server; once the time difference between the system time and the start processing time of the file is greater than the cache file list processing timeout time, return to step (3), thereby removing those files that have timed out and do not need to be transmitted; at the same time, only after the data object to be stored is successfully transmitted, the next cache file is processed. If the data object to be stored is not successfully transmitted, it means that there is an abnormality in the data storage end, and then the data object to be stored is transmitted again after the data storage end returns to normal, thereby achieving efficient and reliable data transmission, solving the problems of system blocking and data loss, and ensuring the stability of system operation and the integrity of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of a flow chart of an efficient data transmission method according to an embodiment of the present invention;

[0038] Figure 2 The figure is a schematic diagram of the structure of an efficient data transmission device according to an embodiment of the present invention.

[0039] Wherein, the accompanying drawings are marked as follows:

[0040] Data generation module 10, cache file module 20, file arrangement module 30, first judgment module 40, file removal module 50, second judgment module 60, file opening module 70, third judgment module 80, and continued processing module 90. DETAILED DESCRIPTION

[0041] The present invention provides the following description with reference to the accompanying drawings to assist in a comprehensive understanding of various embodiments of the present invention as defined by the claims and their equivalents. The description includes various specific details to assist in understanding, but these details should be considered as exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present invention.

[0042] In the description of the present invention, descriptions of orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the drawings and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0043] It will be understood that when one element (eg, a first element) is “connected” to another element (eg, a second element), the element may be directly connected to the other element or an intervening element (eg, a third element) may be present between the element and the other element.

[0044] The embodiment of the present invention provides an efficient data transmission method, such as Figure 1 As shown, the following steps are included:

[0045] S100: Generate a data object to be stored, and put the data object to be stored into a list of data objects to be stored;

[0046] Start the data generation thread, and the data generation end generates the data objects to be stored. The data objects to be stored are the data to be stored in the data storage end such as the database or remote media. These data objects can be different types of data, such as monitoring data, depending on the usage scenario. The data objects to be stored are placed in the list of data objects to be stored for easy subsequent traversal and reference.

[0047] S200: Checking the list of data objects to be stored in real time, and when there is a data object to be stored in the list of data objects to be stored, creating a cache file according to the time series, naming the cache file according to a preset format, and writing the data object to be stored into the corresponding cache file;

[0048] Start the cache file generation thread. The cache files can be newly created in chronological order and named according to a preset format to facilitate unified classification and management of cache files. The preset format can be used to determine whether the file is the required file; each cache file can store one or more groups of data objects to be stored.

[0049] S300: Obtain a local cache file name list, and remove files whose names do not conform to the preset format from the local cache file name list;

[0050] Start the cache file storage thread. Other files may remain in the local cache file name list. In this embodiment, only the cache file containing the data object to be stored needs to be transferred, and other files need to be removed. To this end, whether the name conforms to the preset format can be used to determine whether it is a cache file that needs to be transferred. After removing the files whose names do not conform to the preset format from the local cache file name list, only the files whose names conform to the preset format are retained, avoiding local cache file redundancy and reducing the occupancy of system channels.

[0051] S400: Determine whether the number of files in the local cache file name list is greater than the maximum number of historical cache files;

[0052] The maximum number of historical cache files is the maximum number of cache files that can be saved locally. If the number is greater than this, it will exceed the local load capacity and cause excessive system memory usage.

[0053] S500: If the number of files is greater than the maximum number of historical cache files, remove the files one by one in time sequence starting from the oldest file until the number of files is less than or equal to the maximum number of historical cache files; if the number of files is less than or equal to the maximum number of historical cache files, proceed directly to the subsequent steps;

[0054] The value of the earliest files is very low, and even the usable period has expired. They are useless files. Removing these files will not affect the normal use of the system, and can reduce the number of local cache files, solve the problem of excessive system memory usage, and reduce system congestion. If the number of files is less than or equal to the maximum number of historical cache files, there is no need to remove the files, and proceed directly to the subsequent steps.

[0055] S600: traverse and process all files in the local cache file name list, and determine whether the time difference between the system time and the start time of file processing is greater than the cache file list processing timeout time;

[0056] The cache file list processing timeout is the maximum time that the cache file is waiting to be processed by the system preset. It is necessary to determine whether there is a timed-out file in the local cache file name list.

[0057] If the time difference between the system time and the start processing time of the file is greater than the cache file list processing timeout time, return to step S300; if the time difference between the system time and the start processing time of the file is less than or equal to the cache file list processing timeout time, directly proceed to the subsequent steps;

[0058] After returning to step S300, these cached files that have timed out will be removed in step S500. The files that have timed out may be files that do not need to be transmitted. If they continue to be kept in the cache file list, the number of cached files will increase. By removing these files, the number of cached files can be controlled to prevent the number of cached files from growing indefinitely and affecting the overall operation of the server. If the time difference between the system time and the start processing time of the file is less than or equal to the cache file list processing timeout time, there is no need to return to step S300, but proceed directly to the subsequent steps.

[0059] S700: Open the cache file in the local cache file name list, read the data object to be stored, and transmit the data object to be stored to the data storage end according to the business logic;

[0060] Open the local cache file reading stream according to the cache file name, read the data objects to be stored one by one, and transfer the cache files one by one to the data storage end such as the database or remote media.

[0061] S800: Determine whether the data object to be stored is transmitted successfully.

[0062] There are many ways to determine whether the data object to be stored is successfully transmitted. For example, every time the local cache file is transmitted to the data storage end, the data storage end will feedback the successful transmission information. If the local does not receive the successful transmission information, it means that the data was not successfully transmitted.

[0063] S900: If the data object to be stored is transmitted successfully, continue to process the next cache file until all the data objects to be stored are transmitted successfully; if the data object to be stored is not transmitted successfully, return to step (3).

[0064] Only after the data object to be stored is successfully transmitted, it means that the communication between the local and data storage ends is smooth, and the next cache file can be processed until all data objects to be stored are successfully transmitted and the transmission of the data objects to be stored is completed; if there is a situation where the data object to be stored is not successfully transmitted, it means that there is an abnormality in the data storage end such as the database or the remote medium, then return to step S300, wait for recovery, and then continue to transmit the data object to be stored. The present invention uses multi-threaded asynchronous processing to write the data object to be stored into the cache file first, reduce business system blocking, reduce business system memory usage, and improve business system response speed. It can avoid the problem of data loss caused by the inability to establish communication with the data storage end, realize efficient and reliable data transmission, solve the problem of system blocking and data loss, and ensure the stability of system operation and data integrity.

[0065] In some embodiments, in step (2), if there is no data object to be stored in the list of data objects to be stored, the process continues to wait in step (2) until there is a data object to be stored, and then proceeds to step (3).

[0066] In some embodiments, in step S200, the step of establishing a cache file according to a time series and naming the cache file according to a preset format specifically includes: defining the time when the list of data objects to be stored is obtained as tmBegin, traversing and processing the list of data objects to be stored, defining the subscript of the current data object to be stored as i, i starts from 0, defining the current time of the i-th data object to be stored as tmNow, and assigning tmNow to tmBegin plus i milliseconds. The name of the cache file includes the current time tmNow of the i-th data object to be stored and a preset identification field.

[0067] The complete name of the cache file can be "..\tmNow.Hour\collector ID of data object i to be stored_tmNow_timestamp of data object i to be stored.hi scache", for example, "..\18\100001_202311191807223_20231119180000.hi scache", where tmNow.Hour is the hour in the current time tmNow of the i-th data object to be stored, and hi scache is an identification field used to indicate that this is a cache file for storing data objects to be stored.

[0068] In this embodiment, the local cache file is stored in blocks based on the collector, one folder per hour for 24 hours, to avoid the loss of all data due to disk damage, and to ensure the stability of business system operation and data integrity.

[0069] Furthermore, step (3) specifically includes: periodically obtaining a list of local cache file names, filtering out files whose names contain the identification field, and adding a removal field to the names of files whose name formats do not conform to the preset format, and removing files whose names contain the removal field from the list of local cache file names.

[0070] This embodiment is to obtain valid cache files. Files whose name formats do not conform to the preset format do not belong to the cache files that store the data objects to be stored. A removal field is added to the file names of these files. For example, the removal field can be "de l ete", and then the files whose names contain the removal field are removed from the local cache file name list.

[0071] Furthermore, when there is no cache file containing the identification field in the local cache file name list, the period for obtaining the local cache file name list is extended.

[0072] When there is no cache file containing the identification field in the local cache file name list, it means that there is no cache file that can be transmitted. The period for periodically obtaining the local cache file name list can be extended. For example, the local cache file name list was originally obtained once every second. After the extension, the local cache file name list can be obtained once every five seconds.

[0073] In some embodiments, between naming the cache file according to a preset format and writing the data object to be stored into the corresponding cache file in step S200, the following step is also included: determining whether the created cache file already exists locally; if the created cache file already exists locally, deleting the existing local file.

[0074] Naming the cache file according to the preset format does not guarantee the uniqueness of the named cache file locally. If the name of the file existing locally is the same as the name of the cache file newly created according to the preset format, a conflict will occur. To this end, first determine whether the cache file to be created already exists locally. If the cache file to be created already exists locally, delete the file already existing locally, then create a new cache file, and then execute the step of writing the data object to be stored into the corresponding cache file. If the cache file to be created does not exist locally, directly execute the step of writing the data object to be stored into the corresponding cache file.

[0075] In some embodiments, in step S200, the step of writing the data object to be stored into the corresponding cache file specifically includes: writing the data object to be stored into the cache file according to a defined cache file format, wherein the cache file format includes 41-character identification information, a 4-character version number and business data arranged in sequence.

[0076] The data objects to be stored are stored in a unified preset format to facilitate unified management. The cache files can be stored in binary format. First, the 41-character identification information HisCacheFlag is written, followed by the 4-byte version number HisCacheVersion, followed by business data, such as an 8-byte data packet timestamp, a 4-byte array length, a 4-byte data item 1 identifier, an 8-byte data item 1 value, a 4-byte data item n identifier, and an 8-byte data item n value.

[0077] In some embodiments, the following steps are also included between step S500 and step S600: traverse all cache files in the local cache file name list, compare the current time tmNow of the i-th data object to be stored in the file name with the current time of the system, and when the absolute value of the difference between the two times is less than the time threshold, remove the file from the local cache file name list; when the number of files in the local cache file name list is zero, enter step (3).

[0078] When the absolute value of the difference between the current time tmNow of the i-th data object to be stored in the file name and the current time of the system is less than the time threshold, it means that the generation time of the current file is too recent. For files with too recent time, it is not necessary to transfer them immediately, otherwise the number of cached files will increase. They should be removed from the local cached file name list in time to avoid the problem of excessive system memory usage caused by too large a number of local cached files, thereby reducing system congestion. If the number of files in the local cached file name list is zero, return to step S300 and continue to enable the cached file storage thread.

[0079] In some embodiments, after step (10), the following steps are also included: periodically obtaining a local cache file name list, and deleting files whose names contain a removal field from the local cache file name list.

[0080] After step S900, the cache file deletion thread is enabled to perform a systematic deletion process on the files that have been added or removed in the entire process. Specifically, the file containing the removed field is first searched. After the file containing the removed field is found, the system sleeps for 1 second and then deletes the files containing the removed field one by one. The period for obtaining the local cache file name list can be 60 seconds.

[0081] The embodiment of the present invention provides an efficient data transmission device, such as Figure 2 As shown, including:

[0082] The data generation module 10 is used to generate a data object to be stored and put the data object to be stored into a list of data objects to be stored;

[0083] The cache file module 20 is used to check the list of data objects to be stored in real time. When there is a data object to be stored in the list of data objects to be stored, a cache file is created according to the time series, and the cache file is named according to a preset format, and the data object to be stored is written into the corresponding cache file;

[0084] A file sorting module 30 is used to obtain a local cache file name list and remove files whose names do not conform to a preset format from the local cache file name list;

[0085] A first judgment module 40 is used to judge whether the number of files in the local cache file name list is greater than the maximum number of historical cache files;

[0086] The file removal module 50 is used to remove files one by one in time sequence starting from the earliest file when the first judgment module determines that the number of files is greater than the maximum number of historical cache files, until the number of files is less than or equal to the maximum number of historical cache files;

[0087] The second judgment module 60 is used to traverse and process all the files in the local cache file name list when the first judgment module determines that the number of files is less than or equal to the maximum number of historical cache files, and determine whether the time difference between the system time and the start time of file processing is greater than the cache file list processing timeout time;

[0088] The file arrangement module 30 is also used to obtain a local cache file name list when the time difference between the system time and the start processing time of the file is greater than the cache file list processing timeout time, and remove files whose names do not conform to the preset format from the local cache file name list;

[0089] The file opening module 70 is used to open the cache file in the local cache file name list, read the data object to be stored, and transmit the data object to be stored to the data storage end according to the business logic when the time difference between the system time and the start processing time of the file is less than or equal to the cache file list processing timeout time;

[0090] The third judgment module 80 is used to judge whether the data object to be stored is successfully transmitted;

[0091] A continuing processing module 90 is used to continue processing the next cache file when the data object to be stored is successfully transmitted, until all the data objects to be stored are successfully transmitted;

[0092] The file sorting module 30 is further used to obtain a local cache file name list when the data object to be stored is not successfully transmitted, and remove files whose names do not conform to a preset format from the local cache file name list.

[0093] In some embodiments, the cache file module 20 is further configured to continue waiting when there is no data object to be stored in the list of data objects to be stored.

[0094] In some embodiments, the cache file module 20 is used to define the time when the list of data objects to be stored is obtained as tmBegin, traverse the list of data objects to be stored, define the subscript of the current data object to be stored as i, i starts from 0, define the current time of the i-th data object to be stored as tmNow, and assign tmNow to tmBegin plus i milliseconds; the name of the cache file includes the current time tmNow of the i-th data object to be stored and a preset identification field. In this way, the cache file is established according to the time series, and the cache file is named according to the preset format.

[0095] Furthermore, the file sorting module 30 is used to periodically obtain a list of local cache file names, filter out files whose names contain the identification field, and add a removal field to the names of files whose name formats do not conform to the preset format, and remove files whose names contain the removal field from the local cache file name list.

[0096] Furthermore, the file sorting module 30 is used to extend the period of obtaining the local cache file name list when there is no cache file containing the identification field in the local cache file name list.

[0097] In some embodiments, the cache file module 20 is further used to determine whether the created cache file already exists locally, and if the created cache file already exists locally, delete the existing file locally.

[0098] In some embodiments, the cache file module 20 is further used to write the data object to be stored into the cache file according to a defined cache file format, wherein the cache file format includes 41-character identification information, a 4-character version number and business data arranged in sequence.

[0099] In some embodiments, a time comparison module is also included, which is used to traverse all cache files in the local cache file name list, compare the current time tmNow of the i-th data object to be stored in the file name with the current time of the system, and when the absolute value of the difference between the two times is less than the time threshold, remove the file from the local cache file name list.

[0100] After the number of files in the local cache file name list reaches zero, the file sorting module continues to obtain the local cache file name list, and removes files whose names do not conform to the preset format from the local cache file name list.

[0101] In some embodiments, a periodic acquisition module is further included, which is used to periodically acquire a local cache file name list and delete files whose names contain a removal field from the local cache file name list.

[0102] The efficient data transmission device of this embodiment is a device type embodiment corresponding to the above method part. The specific description of the efficient data transmission device can be referred to the above embodiment, which will not be repeated here.

[0103] The terms and words used in the above description and claims are not limited to the literal meanings, but are only used by the applicant to enable a clear and consistent understanding of the present invention. Therefore, it should be clear to those skilled in the art that the above description of various embodiments of the present invention is provided only for illustration, and not for limiting the present invention as defined by the attached claims and their equivalents.

Claims

1. An efficient data transmission method, characterized in that: The steps include: (1) Generate a data object to be stored and put the data object to be stored into a list of data objects to be stored; (2) checking the list of data objects to be stored in real time, and when there is a data object to be stored in the list of data objects to be stored, creating a cache file according to the time series, naming the cache file according to a preset format, and writing the data object to be stored into the corresponding cache file; (3) obtaining a local cache file name list, and removing files whose names do not conform to the preset format from the local cache file name list; (4) determining whether the number of files in the local cache file name list is greater than the maximum number of historical cache files; (5) If the number of files is greater than the maximum number of historical cache files, remove the files one by one in chronological order, starting from the oldest file, until the number of files is less than or equal to the maximum number of historical cache files; if the number of files is less than or equal to the maximum number of historical cache files, proceed directly to the subsequent steps; (6) traverse and process all files in the local cache file name list to determine whether the time difference between the system time and the start time of file processing is greater than the cache file list processing timeout; (7) If the time difference between the system time and the start time of the file processing is greater than the cache file list processing timeout, return to step (3); if the time difference between the system time and the start time of the file processing is less than or equal to the cache file list processing timeout, directly proceed to the subsequent steps; (8) Open the cache file in the local cache file name list, read the data object to be stored, and transmit the data object to be stored to the data storage end according to the business logic; (9) Determine whether the data object to be stored is successfully transmitted; (10) If the data object to be stored is transmitted successfully, continue to process the next cache file until all the data objects to be stored are transmitted successfully; if the data object to be stored is not transmitted successfully, return to step (3).

2. The efficient data transmission method according to claim 1, characterized in that: In step (2), if there is no data object to be stored in the list of data objects to be stored, continue to wait in step (2).

3. The efficient data transmission method according to claim 1, characterized in that: The steps of establishing a cache file according to a time series and naming the cache file according to a preset format specifically include: defining the moment when a list of data objects to be stored is obtained as tmBegin, traversing and processing the list of data objects to be stored, defining the subscript of the current data object to be stored as i, i starts from 0, defining the current time of the i-th data object to be stored as tmNow, and assigning tmNow to tmBegin plus i milliseconds; the name of the cache file includes the current time tmNow of the i-th data object to be stored and a preset identification field.

4. The efficient data transmission method according to claim 3, characterized in that: Step (3) specifically includes: periodically obtaining a local cache file name list, filtering out files whose names contain the identification field, and adding a removal field to the names of files whose name formats do not conform to the preset format, and removing files whose names contain the removal field from the local cache file name list.

5. The efficient data transmission method according to claim 4, characterized in that: When there is no cache file containing the identification field in the local cache file name list, the period for obtaining the local cache file name list is extended.

6. The efficient data transmission method according to claim 1, characterized in that: Between naming the cache file according to a preset format and writing the data object to be stored into the corresponding cache file, the following steps are also included: determining whether the created cache file already exists locally; if the created cache file already exists locally, deleting the existing local file.

7. The efficient data transmission method according to claim 1, characterized in that: The step of writing the data object to be stored into the corresponding cache file specifically includes: writing the data object to be stored into the cache file according to a defined cache file format, wherein the cache file format includes 41-character identification information, a 4-character version number and business data arranged in sequence.

8. The efficient data transmission method according to claim 3, characterized in that: The following steps are also included between step (5) and step (6): traverse all cache files in the local cache file name list, compare the current time tmNow of the i-th data object to be stored in the file name with the current time of the system, and when the absolute value of the difference between the two times is less than the time threshold, remove the file from the local cache file name list; when the number of files in the local cache file name list is zero, enter step (3).

9. The efficient data transmission method according to claim 4, characterized in that: After step (10), the following step is also included: periodically obtaining a local cache file name list, and deleting files whose names contain the removal field from the local cache file name list.

10. An efficient data transmission device, characterized in that: include: A data generation module, used for generating a data object to be stored, and putting the data object to be stored into a list of data objects to be stored; The cache file module is used to check the list of data objects to be stored in real time. When there is a data object to be stored in the list of data objects to be stored, a cache file is created according to the time series, and the cache file is named according to a preset format, and the data object to be stored is written into the corresponding cache file; A file sorting module is used to obtain a local cache file name list and remove files whose names do not conform to the preset format from the local cache file name list; The first judgment module is used to judge whether the number of files in the local cache file name list is greater than the maximum number of historical cache files; A file removal module, for removing files one by one in time sequence starting from the earliest file when the first judgment module judges that the number of files is greater than the maximum number of historical cache files, until the number of files is less than or equal to the maximum number of historical cache files; The second judgment module is used to traverse and process all the files in the local cache file name list when the first judgment module determines that the number of files is less than or equal to the maximum number of historical cache files, and determine whether the time difference between the system time and the start time of file processing is greater than the cache file list processing timeout time; The file arrangement module is also used to obtain a local cache file name list when the time difference between the system time and the start processing time of the file is greater than the cache file list processing timeout time, and remove files whose names do not conform to the preset format from the local cache file name list; The file opening module is used to open the cache file in the local cache file name list when the time difference between the system time and the start processing time of the file is less than or equal to the cache file list processing timeout time, read the data object to be stored, and transmit the data object to be stored to the data storage end according to the business logic; A third judgment module is used to judge whether the data object to be stored is successfully transmitted; A continuing processing module is used to continue processing the next cache file when the data object to be stored is successfully transmitted, until all the data objects to be stored are successfully transmitted; The file arrangement module is also used to obtain a local cache file name list when the data object to be stored is not successfully transmitted, and remove files whose names do not conform to the preset format from the local cache file name list.

Images