Data storage method and related device for reliable transmission

Abstract

An embodiment of the present application provides a data storage method and related device for reliable transmission, the method comprising: receiving a message to be transmitted for reliable transmission; determining a file header of the message to be transmitted; segmenting the message to be transmitted to obtain K file fragments, the K file fragments having the same length, and determining record end identifiers of the K file fragments; constructing a target file according to a preset file construction method based on the file header of the message to be transmitted, the K file fragments, and the record end identifiers of the K file fragments; when storing or reading the target file, by storing or reading each message to be transmitted at most once, the number of file reading and writing is reduced, and sequential reading and writing is performed, which facilitates batch reading and writing and improves the efficiency of file storage or reading.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and specifically to a data storage method and related apparatus for reliable transmission. Background Technology

[0002] When using network communication for reliable message transmission, messages need to be stored in files to prevent data loss in case of program failure. Currently, when storing general messages with highly variable lengths, it is impossible to determine the business type and length of the transmitted message in advance. The message length can vary greatly, from a few bytes to several megabytes. We cannot store the message bytes in a single record in the database. We must find a file storage format that can quickly store variable-length messages.

[0003] Currently, this problem is generally solved by using multiple file storage. For example, a fixed-length index file can be used, along with a data file. In the index file, each record is fixed-length, and each record contains an offset pointing to the data file. When locating data, the record number is first located in the index file, and then the data, along with other attribute information, is read from the data file based on the offset in the index file. Message queues like Kafka use this model.

[0004] The above methods can solve the problem of reliable file storage, but reading and writing any record requires reading and writing once in the index file and once in the data file. Since it involves reading and writing two files, and each message needs to be read and written separately, the disk head moves at least twice, and it is not a sequential read and write. According to the common sense of computer storage, the disk efficiency is very low when reading and writing files randomly. Summary of the Invention

[0005] This application provides a data storage method and related apparatus for reliable transmission. By storing or retrieving each message to be transmitted at most once, the number of file read / write operations is reduced, and sequential read / write operations improve the efficiency of file storage or retrieval. In this application embodiment, the line number of the record is used as the record identifier for reliable transmission, making it simple and convenient to use.

[0006] A first aspect of this application provides a data storage method for reliable transmission, the method comprising:

[0007] Receive messages to be transmitted for reliable transmission;

[0008] Determine the file header of the message to be transmitted;

[0009] The message to be transmitted is segmented to obtain K file segments of equal length, and the record end identifier of the K file segments is determined.

[0010] The target file is constructed according to the file header of the message to be transmitted, K file segments, and the record end identifiers of the K file segments, using a preset file construction method.

[0011] The target file is stored.

[0012] In one possible implementation, the step of constructing the target file according to a preset file construction method based on the file header of the message to be transmitted, K file segments, and record end identifiers of the K file segments, to obtain the target file includes:

[0013] Obtain the file construction template corresponding to the message to be transmitted;

[0014] The target file is constructed by using the file construction template to construct the file header of the message to be transmitted, K file segments, and the record end identifier of the K file segments.

[0015] In one possible implementation, the step of constructing the target file according to a preset file construction method based on the file header of the message to be transmitted, K file segments, and record end identifiers of the K file segments, to obtain the target file includes:

[0016] Write the file header into the file to obtain the first intermediate file;

[0017] In the first intermediate file, the record end identifier of K file segments of the first message to be transmitted, the effective length of the K file segments, and the content of the K file segments are written line by line in sequence to obtain the second intermediate file. If the message is the last message, the second intermediate file is determined to be the target file.

[0018] Continue repeating the above process, writing subsequent messages sequentially to obtain subsequent intermediate files. If the message is the last message, then the subsequent intermediate file is determined to be the target file.

[0019] In one possible implementation, the segmentation of the message to be transmitted to obtain K file fragments includes:

[0020] Get the preset single-line record length;

[0021] Determine whether the data volume of the message to be transmitted is divisible by the data volume corresponding to the preset single-line record length. If it is divisible, then the message to be transmitted is divided equally to obtain the K file segments.

[0022] If the result is not divisible, the message to be transmitted is divided equally according to the preset single-line record length to obtain K. One file segment, and fixedly fill the non-transmitted data bits in the Kth file segment to obtain the filled Kth file segment.

[0023] In one possible implementation, the method further includes:

[0024] Read the message set to be transmitted;

[0025] The reading of the message set to be transmitted includes:

[0026] Obtain the file header of the message to be transmitted, which includes information about the file header length and the length of a single line record. The line number of the record is the record identifier used for reliable transmission;

[0027] Based on the single-line record length information in the file header, M lines of data are read into memory at once. In memory, K message packets to be transmitted are obtained by disassembling and recovering the record end marker. If the record end marker of the last line of the data read this time is not finished, the tail data of this time is merged with the head data line of the next read from memory to obtain a complete message packet to be transmitted.

[0028] Repeat the above process to obtain a message packets to be transmitted, where a is a positive integer greater than or equal to K.

[0029] A second aspect of this application provides a data storage device for reliable transmission, the device comprising:

[0030] A receiving unit is used to receive messages to be transmitted for reliable transmission;

[0031] The determining unit is used to determine the file header of the message to be transmitted;

[0032] The processing unit is used to segment the message to be transmitted to obtain K file segments, the K file segments having the same length, and to determine the record end identifier of the K file segments;

[0033] The construction unit is used to construct the target file according to a preset file construction method based on the file header of the message to be transmitted, K file segments, and the record end identifiers of the K file segments;

[0034] A storage unit is used to store the target file.

[0035] In one possible implementation, the building unit is used for:

[0036] Obtain the file construction template corresponding to the message to be transmitted;

[0037] The target file is constructed by using the file construction template to construct the file header of the message to be transmitted, K file segments, and the record end identifier of the K file segments.

[0038] In one possible implementation, the building unit is used for:

[0039] Write the file header into the file to obtain the first intermediate file;

[0040] In the first intermediate file, the record end identifiers of K file segments of the first message to be transmitted, the effective lengths of the K file segments, and the contents of the K file segments are written line by line sequentially to obtain the second intermediate file. If it is the last message, it is the target file.

[0041] Continue repeating the above process, writing subsequent messages sequentially to obtain the intermediate files. The last message is the target file.

[0042] In one possible implementation, regarding the segmentation of the message to be transmitted to obtain K file fragments, the processing unit is configured to:

[0043] Get the preset single-line record length;

[0044] Determine whether the data volume of the message to be transmitted is divisible by the data volume corresponding to the preset single-line record length. If it is divisible, then the message to be transmitted is divided equally to obtain the K file segments.

[0045] If the result is not divisible, the message to be transmitted is divided equally according to the preset single-line record length to obtain K. One file segment, and fixedly fill the non-transmitted data bits in the Kth file segment to obtain the filled Kth file segment.

[0046] In one possible implementation, the device is further used for:

[0047] Read the message set to be transmitted;

[0048] The reading of the message set to be transmitted includes:

[0049] Obtain the file header of the message to be transmitted, wherein the file header includes information on the file header length and the length of a single line record;

[0050] Based on the single-line record length information in the file header, M lines of data are read into memory at once. In memory, K message packets to be transmitted are obtained by disassembling and recovering the record end marker. If the record end marker of the last line of the data read this time is not finished, the tail data of this time is merged with the head data line of the next read from memory to obtain a complete message packet to be transmitted.

[0051] Repeat the above process to obtain a message packets to be transmitted, where a is a positive integer greater than or equal to K.

[0052] A third aspect of this application provides a terminal including a processor, an input device, an output device, and a memory, wherein the processor, input device, output device, and memory are interconnected, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to execute the step instructions as described in the first aspect of this application.

[0053] A fourth aspect of this application provides a computer-readable storage medium storing a computer program for electronic data interchange, wherein the computer program causes a computer to perform some or all of the steps described in the first aspect of this application.

[0054] A fifth aspect of this application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in the first aspect of this application. The computer program product may be a software installation package.

[0055] Implementing the embodiments of this application has at least the following beneficial effects:

[0056] By receiving a message to be transmitted for reliable transmission, determining the file header of the message to be transmitted, segmenting the message to be transmitted into K file segments of equal length, and determining the record end identifiers of the K file segments, and constructing a target file according to a preset file construction method based on the file header, the K file segments, and the record end identifiers of the K file segments, the target file is then stored. Therefore, by storing the target file constructed from the message to be transmitted in a single, efficient sequential read / write operation, the number of file storage operations can be reduced, thus improving the efficiency of file storage. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1A This application provides a flowchart illustrating a data storage method for reliable transmission.

[0059] Figure 1B A schematic diagram of an object file is provided for an embodiment of this application;

[0060] Figure 2 This application provides a flowchart illustrating another data storage method for reliable transmission.

[0061] Figure 3 This application provides a schematic diagram of the structure of a server according to an embodiment of the present application.

[0062] Figure 4 This application provides a schematic diagram of the structure of a data storage device for reliable transmission. Detailed Implementation

[0063] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0064] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0065] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0066] To better understand the data storage method for reliable transmission provided in this application embodiment, the scenarios for such data storage methods are briefly described below. When using network communication for reliable message transmission, messages need to be stored in files to prevent data loss due to program malfunctions. This can be categorized into two types:

[0067] (1) For messages with very small length changes or fixed lengths.

[0068] Messages with minimal length variation typically have clear business meanings. Generally, the business fields corresponding to each message are stored in a database as records to support reliable transmission.

[0069] (2) For general messages with large variations in length.

[0070] For messages (data to be transmitted) in software similar to message brokers, the message type and length cannot be determined in advance, resulting in a wide variation in message length, ranging from a few bytes to several megabytes. This makes it impractical to store the entire message in a single database record. Currently, the common approach to storing general-purpose messages with varying lengths is to use multiple files. For example, a fixed-length index file and a data file can be used. In the index file, each record is fixed-length, and each record contains an offset pointing to the data file. When locating data, the record number is first located in the index file, and then the data, along with other attribute information, is read from the data file based on the offset in the index file. Message brokers like Kafka use this model.

[0071] The above methods can solve the file storage problem for reliable transmission. However, reading and writing any record requires reading and writing once in the index file and once in the data file. Although each file read and write is sequential, because it involves reading and writing two files and each message needs to be read separately, batch reading is not possible, increasing the number of times the disk head moves (even if a lot of data is read into memory at once, each record still needs to be split in a relatively complex way, and the mapping calculation with the file offset is required). Disk read and write is a very time-consuming operation compared to memory read, especially since the disk head's positioning from the index file to the data file is random and consumes a lot of time, resulting in low efficiency in data storage or reading. Therefore, the embodiments of this application can store the target file of the message to be transmitted in one go, and use a more efficient sequential read and write method, which can reduce the number of file storage operations and improve the efficiency of file storage.

[0072] See also Figure 1A , Figure 1A This application provides a flowchart illustrating a data storage method for reliable transmission. Figure 1A As shown, this method can be applied to a server, and the method includes:

[0073] 101. Receive messages to be transmitted for reliable transmission.

[0074] When reliably transmitting files, the message to be transmitted can be a message sent from the client, a message sent from another server, or a message sent from another device for reliable transmission.

[0075] 102. Determine the file header of the message to be transmitted.

[0076] It can obtain information such as the version number of the message to be transmitted, as well as the length of the file header and the preset single-line record length. The file header length and the preset single-line record length can be set using historical data or empirical values. Therefore, the file header can be determined based on the version number of the message to be transmitted, the file header length, and the preset single-line record length.

[0077] 103. The message to be transmitted is segmented to obtain K file segments, the K file segments have the same length, and the record end identifier of the K file segments is determined.

[0078] Except for the last segment, which has "Yes" for "End of Recording", all other recorded segments have "No".

[0079] When segmenting the message to be transmitted, it can be done according to the length corresponding to a preset single-line record length, resulting in K file segments. An end-of-record identifier is then determined for each file segment. This identifier indicates whether the file segment is the last file segment of the message to be transmitted. For example, the end-of-record identifier is 0 or 1, where 1 indicates the record is not finished, and 0 indicates the record ends at this segment (i.e., the last segment of the message to be transmitted). The segment is represented in binary, with a fixed length of 1 byte. The actual length of the last file segment among the K file segments can be less than the length of the file segment itself. The actual length of the last segment can be understood as the length containing data, since the last segment may contain padding (when the length of the message to be transmitted is not divisible by the length of the file segment).

[0080] 104. Construct the target file according to the file header of the message to be transmitted, K file segments, and the record end identifiers of the K file segments using a preset file construction method.

[0081] This can be achieved by obtaining a file construction template corresponding to the message to be transmitted, and then constructing the target file based on that template. Alternatively, the target file can be obtained by writing an intermediate file into memory.

[0082] After the target file is constructed, subsequent messages to be transmitted can be written into the target file, thereby enabling the storage of multiple messages to be transmitted through a single file.

[0083] 105. Store the target file.

[0084] The target file can be written to all at once in an append-only manner, or multiple target files containing messages to be transmitted can be cached and written all at once after a certain number of records have been cached or a certain time interval has elapsed. Since message middleware such as Kafka requires writing to the index file and data file at least once each, this method reduces the number of writes and greatly improves storage efficiency compared to traditional message storage methods.

[0085] If other messages to be transmitted are received subsequently, the message can be processed in accordance with the method for determining the target file described above, and the file determined by the message can be appended to the target file. When appending, the files can be appended in chronological order. During the appending process, the file header also needs to be updated to append the relevant information of subsequent messages to the file header.

[0086] In this example, by receiving a message to be transmitted for reliable transmission, determining the file header of the message to be transmitted, segmenting the message to be transmitted into K file segments of equal length, and determining the record end identifiers of the K file segments, a target file is constructed according to a preset file construction method based on the file header of the message to be transmitted, the K file segments, and the record end identifiers of the K file segments. The target file is then stored. Therefore, by storing the target file constructed from the message to be transmitted in a single, efficient sequential read / write operation, the number of file storage operations can be reduced, improving the efficiency of file storage.

[0087] In one possible implementation, a method for constructing a target file according to a preset file construction method based on the file header of the message to be transmitted, K file segments, and record end identifiers of the K file segments, includes:

[0088] A1. Obtain the file construction template corresponding to the message to be transmitted;

[0089] A2. Construct the target file by using the file construction template to construct the file header of the message to be transmitted, K file segments, and the record end identifier of the K file segments.

[0090] The file construction template can be determined based on the type of message to be transmitted. Different types correspond to different file construction templates. The types of messages to be transmitted can include fixed-length and non-fixed-length types. A fixed-length type can be understood as multiple messages to be transmitted having the same data size, or the difference between their data sizes being small, such as a difference less than a fixed threshold. A non-fixed-length type can be understood as multiple messages to be transmitted having a larger difference in data size, such as a difference greater than a fixed threshold. This fixed threshold is set based on empirical values ​​or historical data.

[0091] The target file can be obtained by constructing the file header, K file fragments, and the K file fragments of the message to be transmitted according to the construction method shown in the file construction template.

[0092] In this example, a file can be quickly built using a file building template to obtain the target file, thus improving efficiency when determining the target file.

[0093] In one possible implementation, another possible method for constructing the target file according to a preset file construction method based on the file header of the message to be transmitted, K file segments, and record end identifiers of the K file segments, includes:

[0094] B1. Write the file header into the file to obtain the first intermediate file;

[0095] B2. In the first intermediate file, the record end identifier of K file segments of the first message to be transmitted, the effective length of the K file segments, and the content of the K file segments are written line by line in sequence to obtain the second intermediate file. If the message is the last message, the second intermediate file is determined to be the target file.

[0096] B3. Continue to repeat the above process, writing subsequent messages sequentially to obtain subsequent intermediate files. If the message is the last message, then the subsequent intermediate file is determined to be the target file.

[0097] First, a file header is written. The file header may contain information such as the file version number, header length, and the length of each line of record, which can be defined by the user, thus obtaining the first intermediate file.

[0098] Organize the data to be written to the file in memory. For each file segment, perform the following loop:

[0099] First, write the end-of-record marker. If this is the last file segment of this business message, write 0 at this marker; otherwise, write 1.

[0100] Next, write the valid length of the data in this segment. If this is the last segment of this business message, write the actual valid length of the data in this segment in this field; otherwise, write the fixed length of the segment. This valid length can be understood as the length of the data stored in the file segment. If it is not the last segment, the valid length of the data can be understood as the length of the segment.

[0101] like Figure 1B As shown, Figure 1B A schematic diagram of an object file is shown. Figure 1B The fragment data in the file corresponds to the file fragment, the record end marker corresponds to the record end identifier, and the valid fragment length corresponds to the valid length of the file fragment.

[0102] In one possible implementation, a possible method for segmenting the message to be transmitted to obtain K file fragments includes:

[0103] C1. Get the preset single-line record length;

[0104] C2. Determine whether the data volume of the message to be transmitted can be divided by the data volume corresponding to the preset single-line record length. If it can be divided, then the message to be transmitted is divided equally to obtain the K file segments.

[0105] C3. If the result is not divisible, the message to be transmitted is divided equally according to the preset single-line record length to obtain K. One file segment, and fixedly fill the non-transmitted data bits in the Kth file segment to obtain the filled Kth file segment.

[0106] The preset single-line record length can be set using empirical values ​​or historical data.

[0107] When not divisible, the first K The length of a file segment is the length specified by the preset single-line record length, K. The data in one file segment is the data to be transmitted. The Kth file segment contains part of the data to be transmitted and part of the padding data for fixed padding, such as "0". The padding data can be located after the data to be transmitted, or at other locations.

[0108] In one possible implementation, the message to be transmitted can also be read, and the specific reading method is as follows:

[0109] Read the message set to be transmitted. Specifically, the method for reading the message set to be transmitted includes:

[0110] D1. Obtain the file header of the message to be transmitted. The file header includes information about the file header length and the length of a single line record. The line number of the record is the record identifier used for reliable transmission.

[0111] D2. Based on the single-line record length information in the file header, read M lines of data (M can be 100 to 10000, depending on the memory size) into memory at once. In memory, disassemble and recover K message packets to be transmitted according to the record end marker. If the record end marker of the last line of the data read this time is not finished, merge the tail data of this time with the header data line of the next read from memory to obtain a complete message packet to be transmitted.

[0112] D3. Repeat the above process to obtain a message packets to be transmitted, where a is a positive integer greater than or equal to K.

[0113] When reading data, multiple messages to be transmitted (i.e., a set of messages to be transmitted) can be read at once. During reading, the file header can be read first from the storage. The file header includes information such as the file header length and the length of a single line record.

[0114] For example, if the preset single-line record length is 500 bytes, then the message to be transmitted can be read in 200,000 lines. Each line includes a file segment, a record end marker, and the valid length of the file segment. Therefore, compared to existing solutions that can only read data corresponding to one message to be transmitted at a time, this greatly reduces data reading time and improves reading efficiency. Specifically, sequential read and write operations can be implemented during data reading.

[0115] The data storage method of this application embodiment can (1) reduce the number of file read / write operations and achieve sequential read / write, thereby improving the efficiency of file read / write. Especially when reading files, the disk operation that previously required reading each record twice can be reduced to reading only once for every 200,000 file records, greatly improving disk read performance. (This calculation is based on a record length of 500 bytes, which means 200,000 file records can be read at a time. If the parameters are different, the number of disk reads will be different.)

[0116] (2) All information is stored in one file, thus reducing the number of times the read / write head moves, improving storage performance, and making the interface simpler and file organization and movement more convenient.

[0117] (3) Separating the two concepts of the length of the business layer message packet and the maximum packet length that can be sent each time in reliable transmission reflects the design idea of ​​layering the business layer and the transport layer. At the same time, this design can support scenarios where the transport layer has a length limit for each data transmission (such as UDP transmission, message middleware systems with a maximum length limit for a single packet, etc.), and has wider adaptability.

[0118] See also Figure 2 , Figure 2 This application provides a flowchart illustrating a data storage method for reliable transmission. Figure 2 As shown, this method can be applied to a server, and the method includes:

[0119] 201. Receive the message to be transmitted for reliable transmission;

[0120] 202. Determine the file header of the message to be transmitted;

[0121] 203. The message to be transmitted is segmented to obtain K file segments, the K file segments have the same length, and the record end identifier of the K file segments is determined;

[0122] 204. Construct the target file according to the file header of the message to be transmitted, K file segments, and the record end identifier of the K file segments using a preset file construction method;

[0123] 205. Store the target file;

[0124] 206. Obtain the file header of the message to be transmitted, wherein the file header includes information on the file header length and the length of a single line record. The line number of the record is the record identifier used for reliable transmission;

[0125] 207. Based on the single-line record length information in the file header, read M lines of data into memory at once. In memory, disassemble and recover K message packets to be transmitted according to the record end mark. If the record end mark of the last line of the data read this time is not finished, merge the tail data of this time with the head data line of the next read from memory to obtain a complete message packet to be transmitted.

[0126] 208. Repeat the above process to obtain a message packets to be transmitted, where a is a positive integer greater than or equal to K.

[0127] In this example, multiple messages to be transmitted can be read at once. Compared with the existing solution, which can only read the data corresponding to one message to be transmitted at a time, this can greatly reduce the data reading time and improve the reading efficiency.

[0128] For examples consistent with the above embodiments, please refer to... Figure 3 , Figure 3 A schematic diagram of a server structure provided in an embodiment of this application is shown in the figure. It includes a processor, an input device, an output device, and a memory. The processor, input device, output device, and memory are interconnected. The memory is used to store a computer program, which includes program instructions. The processor is configured to call the program instructions. The program includes instructions for performing the following steps.

[0129] Receive messages to be transmitted for reliable transmission;

[0130] Determine the file header of the message to be transmitted;

[0131] The message to be transmitted is segmented to obtain K file segments of equal length, and the record end identifier of the K file segments is determined.

[0132] The target file is constructed according to the file header of the message to be transmitted, K file segments, and the record end identifiers of the K file segments, using a preset file construction method.

[0133] The target file is stored.

[0134] The above mainly describes the solutions of the embodiments of this application from the perspective of the method execution process. It is understood that, in order to achieve the above functions, the terminal includes the corresponding hardware structure and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments provided herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0135] The embodiments of this application can divide the terminal into functional units according to the above method examples. For example, each function can be divided into a functional unit, or two or more functions can be integrated into one processing unit.

[0136] The integrated units described above can be implemented in hardware or as software functional units. It should be noted that the unit division in this embodiment is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used.

[0137] For those consistent with the above, please refer to Figure 4 , Figure 4 This application provides a schematic diagram of a data storage device for reliable transmission. For example... Figure 4 As shown, the device includes:

[0138] The receiving unit 401 is used to receive a message to be transmitted for reliable transmission;

[0139] Determining unit 402 is used to determine the file header of the message to be transmitted;

[0140] The processing unit 403 is used to segment the message to be transmitted to obtain K file segments, the K file segments having the same length, and to determine the record end identifier of the K file segments;

[0141] The construction unit 404 is used to construct the target file according to the file header of the message to be transmitted, K file segments, and the record end identifier of the K file segments according to a preset file construction method.

[0142] Storage unit 405 is used to store the target file.

[0143] In one possible implementation, the building unit 404 is used for:

[0144] Obtain the file construction template corresponding to the message to be transmitted;

[0145] The target file is constructed by using the file construction template to construct the file header of the message to be transmitted, K file segments, and the record end identifier of the K file segments.

[0146] In one possible implementation, the building unit 404 is used for:

[0147] Write the file header into the file to obtain the first intermediate file;

[0148] In the first intermediate file, the record end identifier of K file segments of the first message to be transmitted, the effective length of the K file segments, and the content of the K file segments are written line by line in sequence to obtain the second intermediate file. If the message is the last message, the second intermediate file is determined to be the target file.

[0149] Continue repeating the above process, writing subsequent messages sequentially to obtain subsequent intermediate files. If the message is the last message, then the subsequent intermediate file is determined to be the target file.

[0150] In one possible implementation, regarding the segmentation of the message to be transmitted to obtain K file fragments, the processing unit 403 is configured to:

[0151] Get the preset single-line record length;

[0152] Determine whether the data volume of the message to be transmitted is divisible by the data volume corresponding to the preset single-line record length. If it is divisible, then the message to be transmitted is divided equally to obtain the K file segments.

[0153] If the result is not divisible, the message to be transmitted is divided equally according to the preset single-line record length to obtain K. One file segment, and fixedly fill the non-transmitted data bits in the Kth file segment to obtain the filled Kth file segment.

[0154] In one possible implementation, the device is further used for:

[0155] Read the message set to be transmitted;

[0156] The reading of the message set to be transmitted includes:

[0157] Obtain the file header of the message to be transmitted, which includes information about the file header length and the length of a single line record. The line number of the record is the record identifier used for reliable transmission;

[0158] Based on the single-line record length information in the file header, M lines of data are read into memory at once. In memory, K message packets to be transmitted are obtained by disassembling and recovering the record end marker. If the record end marker of the last line of the data read this time is not finished, the tail data of this time is merged with the head data line of the next read from memory to obtain a complete message packet to be transmitted.

[0159] Repeat the above process to obtain a message packets to be transmitted, where a is a positive integer greater than or equal to K.

[0160] This application also provides a computer storage medium that stores a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the data storage methods for reliable transmission described in the above method embodiments.

[0161] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program that causes a computer to perform some or all of the steps of any of the data storage methods for reliable transmission described in the above method embodiments.

[0162] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0163] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0164] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0165] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0166] Furthermore, the functional units in the various embodiments of the application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software program module.

[0167] If the integrated unit is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0168] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include: a flash drive, a read-only memory, a random access memory, a magnetic disk, or an optical disk, etc.

[0169] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data storage method for reliable transmission, characterized in that, The method includes: Receive messages to be transmitted for reliable transmission; Determine the file header of the message to be transmitted; The message to be transmitted is segmented to obtain K file segments of equal length, and the record end identifier of the K file segments is determined. The target file is constructed according to the file header of the message to be transmitted, K file segments, and the record end identifiers of the K file segments, using a preset file construction method. The target file is stored; The process of constructing the target file according to a preset file construction method based on the file header of the message to be transmitted, K file segments, and the record end identifiers of the K file segments, to obtain the target file includes: Write the file header into the file to obtain the first intermediate file; In the first intermediate file, the record end identifier of K file segments of the first message to be transmitted, the effective length of the K file segments, and the content of the K file segments are written line by line in sequence to obtain the second intermediate file. If the message is the last message, the second intermediate file is determined to be the target file. Continue repeating the above process, writing subsequent messages sequentially to obtain subsequent intermediate files. If the message is the last message, then the subsequent intermediate file is determined to be the target file.

2. The method according to claim 1, characterized in that, The segmentation of the message to be transmitted to obtain K file fragments includes: Get the preset single-line record length; Determine whether the data volume of the message to be transmitted is divisible by the data volume corresponding to the preset single-line record length. If it is divisible, then the message to be transmitted is divided equally to obtain the K file segments. If the data cannot be divided evenly, the message to be transmitted is divided into K-1 file segments according to the preset single-line record length, and the data bits of the non-message to be transmitted in the Kth file segment are fixedly filled to obtain the filled Kth file segment.

3. The method according to claim 2, characterized in that, The method further includes: Read the message set to be transmitted; Reading the message set to be transmitted includes: obtaining the file header of the message to be transmitted, wherein the file header includes information on the file header length and the length of a single line record; Based on the single-line record length information in the file header, M lines of data are read into memory at once. In memory, K message packets to be transmitted are obtained by disassembling and recovering the record end marker. If the record end marker of the last line of the data read this time is not finished, the tail data of this time is merged with the head data line of the next read from memory to obtain a complete message packet to be transmitted. Repeat the above process to obtain a message packets to be transmitted, where a is a positive integer greater than or equal to K.

4. A data storage device for reliable transmission, characterized in that, The device includes: A receiving unit is used to receive messages to be transmitted for reliable transmission; The determining unit is used to determine the file header of the message to be transmitted; The processing unit is used to segment the message to be transmitted to obtain K file segments, the K file segments having the same length, and to determine the record end identifier of the K file segments; The construction unit is used to construct the target file according to a preset file construction method based on the file header of the message to be transmitted, K file segments, and the record end identifiers of the K file segments; A storage unit for storing the target file; The building unit is used for: Write the file header into the file to obtain the first intermediate file; In the first intermediate file, the record end identifier of K file segments of the first message to be transmitted, the effective length of the K file segments, and the content of the K file segments are written line by line in sequence to obtain the second intermediate file. If the message is the last message, the second intermediate file is determined to be the target file. Continue repeating the above process, writing subsequent messages sequentially to obtain subsequent intermediate files. If the message is the last message, then the subsequent intermediate file is determined to be the target file.

5. A terminal, characterized in that, The system includes a processor, an input device, an output device, and a memory, which are interconnected. The memory is used to store a computer program, which includes program instructions. The processor is configured to invoke the program instructions to perform the method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-3.

Images