A packet packet write alignment method, system, device and medium

Abstract

The application discloses a packet package write alignment method, system, device and medium, applied to the field of data transmission, acquires a packet package sent by a client, judges whether a current underbrush batch exceeds an underbrush threshold after the packet package is added to the current underbrush batch, if yes, logically divides the packet package and marks that the packet package has been divided, adds a part of the packet package smaller than the underbrush threshold to the current underbrush batch, and adds a remaining part to a next underbrush batch, if not, marks the packet package as not being divided, adds the packet package to the current underbrush batch, judges whether the current underbrush batch added to a waiting underbrush queue satisfies an underbrush condition, if yes, underbrushes the current underbrush batch. According to the underbrush threshold, the packet package is divided and processed, the part not meeting the underbrush threshold is processed in batches, the speed of writing into a disk and reading data is improved, and the occurrence of data garbled codes and errors is reduced.

Description

Technical Field

[0001] This application relates to the field of data transmission, and in particular to a packet writing alignment method, system, apparatus and medium. Background Technology

[0002] With the development of technology, the distributed file system (Hadoop Distributed File System, HDFS) implemented by the distributed system infrastructure establishes a connection with the data node when writing data. The client divides the storage and processing unit (Block) in the database into several data packets (packets) smaller than Blocks and sends them to the server. The packets contain the sequence number, writing position, and writing data length of the data packets for the server to refresh and upgrade.

[0003] Currently, when a client sends a packet to a server, in many real-world application scenarios, the write positions in the packet are not aligned. This can cause the same packet to be written to multiple storage objects when writing to disk, increasing the disk write time during addressing and even causing garbled characters. Similarly, it will also increase the read time and may cause errors.

[0004] How to align packets before writing data to disk when writing data via HDFS is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a packet write alignment method that can solve the problem that when writing data through HDFS, the packet is not aligned when it is flushed and written to the disk.

[0006] To address the aforementioned technical problems, this application provides a packet write alignment method, comprising:

[0007] Retrieve the packet sent by the client;

[0008] Determine whether the current batch exceeds the refresh threshold after the packet is added to the current batch.

[0009] If the threshold is exceeded, the packet is logically split and marked as split. The portion of the packet smaller than the threshold is added to the current batch, and the remaining portion is added to the next batch.

[0010] If the number of packets does not exceed the limit, the packet is marked as unsplitled and added to the current batch.

[0011] Determine whether the current batch meets the conditions for being refreshed;

[0012] If the conditions are met, the current batch is added to the waiting batch queue.

[0013] Preferably, the step of logically splitting the packet includes:

[0014] The portion of the packet that exceeds the brush threshold is marked as a segmented portion.

[0015] Preferably, determining whether the current batch meets the batch conditions includes:

[0016] Get the largest packet size in the current batch.

[0017] Determine whether the sequence number of the packet and its offset in the data block are both 0;

[0018] If all values ​​are 0, the packet is added to the waiting queue for the next flush, and a new flush batch is used as the next flush batch.

[0019] Determine whether the packet has been split;

[0020] If the packet has been segmented, the portion of the packet smaller than the preset down-flush threshold is added to the current down-flush batch, the segmented portion is added to the next down-flush batch, and it is determined whether the data length of the packet is less than the maximum packet size in the down-flush batch in which it is located.

[0021] If it is smaller, then the packet will be added to the next batch.

[0022] If it is not smaller than, then add the packet to the current batch.

[0023] If it has not been split, the packet will be added to the current batch.

[0024] Preferably, after determining that the packet's sequence number and its offset within the data block are not both 0, the method further includes:

[0025] Determine whether the data length of the packet is less than the size of the largest packet in the next batch.

[0026] If the value is less than the specified value, then proceed to the step of adding the packet to the waiting queue for the next flush and replacing it with a new flush batch as the next flush batch.

[0027] Preferably, after determining that the data length of the packet is not less than the size of the largest packet in the next batch, the method further includes:

[0028] Determine whether the packet has been split;

[0029] If the packet has already been segmented, the portion of the packet smaller than the preset refresh threshold is added to the waiting refresh queue, and the segmented portion is added to the next refresh batch.

[0030] Determine whether the packet has been split;

[0031] If it has not been split, then the packet is added to the current batch.

[0032] If it has already been segmented, then the segmented portion is added to the next batch.

[0033] Preferably, after determining that the packet has not been split, the method further includes:

[0034] Determine whether the current batch containing the packet exceeds the batch threshold;

[0035] If the threshold is exceeded, the portion of the current batch that exceeds the threshold is added to the waiting batch queue, the remaining portion is added to the next batch, and the process proceeds to the step of determining whether the packet has been split.

[0036] Preferably, after determining that the current batch containing the packet is less than the batch threshold, the method further includes:

[0037] Determine whether the packet is the last packet in the data block it belongs to;

[0038] If it is the last packet, then the packet is added to the waiting queue for the next flush, and a new flush batch is used as the next flush batch, and the process proceeds to the step of determining whether the packet has been split.

[0039] If it is not the last packet, then the packet is added to the current batch.

[0040] Preferably, after adding the current batch to the waiting batch queue, the method further includes:

[0041] Determine whether the packet is not the last packet.

[0042] If it is not the last packet, then the first batch of data is obtained from the waiting batch of data, and it is determined whether the last packet in the first batch of data is the last packet in the data block.

[0043] If it is the last one, then determine whether there is only one packet in the first batch of downloads;

[0044] If there is only one, add the first batch of data to the confirmation set and wait for the client to confirm. Return the confirmation to the client, remove the first batch of data from the waiting queue, and return to the step of determining whether the packet is not the last packet.

[0045] If there is more than one, the first batch is added to the confirmation set and awaits client confirmation. After the first batch is completed, the process proceeds to remove the first batch from the waiting batch queue and returns to the step of determining whether the packet is not the last packet.

[0046] If it is not the last one, then proceed to the step of adding the first batch of downloads to the confirmation set and waiting for the client to confirm. After the first batch of downloads is completed, proceed to the step of removing the first batch of downloads from the waiting queue.

[0047] Preferably, after determining that the packet is the last packet, the method further includes:

[0048] Determine whether the queue waiting for the next flush is not empty;

[0049] If it is not empty, proceed to the step of obtaining the first batch of data from the waiting batch and determining whether the last packet in the first batch is the last packet in the data block.

[0050] To address the aforementioned technical problems, this application also provides a packet write alignment system, comprising:

[0051] The acquisition module is used to retrieve packets sent by the client.

[0052] The first judgment module is used to determine whether the current batch exceeds the brush threshold after the packet is added to the current batch.

[0053] The first segmentation module is used to logically segment the packet if the threshold is exceeded and mark it as already segmented, add the portion of the packet smaller than the down-flush threshold to the current down-flush batch, and add the remaining portion to the next down-flush batch;

[0054] The second segmentation module is used to mark the packet as unsegmented if the number of segments does not exceed a certain threshold, and to add the packet to the current batch.

[0055] The second judgment module is used to determine whether the current batch meets the brushing conditions.

[0056] The add module is used to add the current batch to the waiting batch queue.

[0057] To address the aforementioned technical problems, this application also provides a packet write alignment device, including a memory for storing computer programs;

[0058] A processor, used to implement the packet write alignment method as described above when executing the computer program.

[0059] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the packet write alignment method described above.

[0060] The packet write alignment method provided in this application specifically involves obtaining the packet sent by the client, determining whether the current flush batch exceeds the flush threshold after adding the packet, and if it does. If it does, the packet is logically split and marked as split, with the portion smaller than the flush threshold added to the current flush batch, and the remaining portion added to the next flush batch. If the packet does not exceed the threshold, it is marked as unsplit and added to the current flush batch. The method then checks whether the current flush batch, already in the flush queue, meets the flush conditions; if so, the current flush batch is flushed. This achieves packet write alignment by splitting the packet to conform to the flush size, and batching portions that do not conform to the flush size, thus improving disk write speed, reducing the possibility of garbled characters, reducing data read time, and minimizing errors. Attached Figure Description

[0061] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 A flowchart illustrating the packet write alignment method provided in this application embodiment;

[0063] Figure 2 A flowchart for determining whether the current batch meets the conditions for brushing is provided in the embodiments of this application;

[0064] Figure 3 This is a flowchart illustrating the waiting queue processing provided in an embodiment of this application;

[0065] Figure 4 This is a flowchart illustrating packet processing, downloading, and confirmation provided in the embodiments of this application;

[0066] Figure 5 This is a structural diagram of the packet write alignment system provided in the embodiments of this application;

[0067] Figure 6 This is a structural diagram of a packet write alignment device provided in another embodiment of this application. Detailed Implementation

[0068] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0069] The purpose of this application is to provide a packet write alignment method that can solve the problem that when writing data through HDFS, the packet is not aligned when it is flushed and written to the disk.

[0070] The packet writing alignment method provided in this application implements the packet splitting, alignment, and brushing operations by a server-side controller, such as a microcontroller unit (MCU). Of course, it can also be implemented by other controllers besides MCU, and this application does not limit it.

[0071] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0072] Figure 1 A flowchart of the packet write alignment method provided in the embodiments of this application is shown below. Figure 1 As shown, the method includes:

[0073] S10: Retrieve the packet sent by the client.

[0074] Specifically, in this embodiment, the client sends a Block to the data node. The Block is then divided into several packets and sent to the server. Each packet records the sequence number (seqno), write position (offset), and write data length (length). The default packet size is 64KB. For example, a 1MB Block can be divided into 16 64KB packets and transmitted to the server. The server can begin storage after receiving the packets containing upgrade information. It should be noted that the packet and Block sizes mentioned above are for illustrative purposes and do not represent the only possible implementation. This embodiment does not impose any specific limitations on the packet or Block sizes.

[0075] As we can see, by dividing a larger block into smaller packets, data can be processed in batches, making it more convenient and faster. This also prevents the failure of an entire block from causing the entire process to fail.

[0076] S11: Determine whether the current batch exceeds the batch threshold after adding the packet. If it does, proceed to step S12; otherwise, proceed to step S13.

[0077] Specifically, in this embodiment, the MCU determines whether adding the current packet to the current batch will exceed the batch threshold. The current batch has a pre-set batch threshold. When this threshold is exceeded, it means the current batch is full, and a new batch needs to be used to receive packets. Allocating packets according to the batch threshold ensures packet alignment, improves disk write speed, and reduces the possibility of garbled characters. For example, if the preset batch threshold is 4MB, and the acquired packet size is 5MB, exceeding the threshold, the first 4MB of the packet needs to be stored in the current batch. Since the current batch is full, the remaining 1MB is stored in the next batch. The next batch includes the last 1MB of the packet, and its remaining 3MB of storage space is used to compare the size of the next acquired packet. If the acquired packet is 1MB and the current batch has more than 1MB of storage space, the current packet can be directly stored in the current batch. It should be noted that the swipe threshold and packet size are intended to help those skilled in the art better understand the implementation, and do not represent that this is the only implementation method. This embodiment does not impose any special limitations on the packet size or the swipe threshold.

[0078] It is evident that by setting a flush threshold to limit the alignment size, the received packets can be flushed in batches according to the flush threshold, which can reduce garbled characters and improve disk write speed.

[0079] S12: Logically split the packet and mark it as split. Add the portion of the packet smaller than the downsweep threshold to the current downsweep batch, and add the remaining portion to the next downsweep batch.

[0080] Specifically, in this embodiment, if the packet is larger than the remaining storage size in the current batch, the packet needs to be logically split. It should be noted that logical splitting is not the same as traditional splitting. Due to the special nature of packets, in order to ensure that data is not lost, the portion of the packet that exceeds the batch threshold needs to be marked as the split portion. However, at this time, the packet is still a whole, and the data loss will not be caused by packet splitting due to logical splitting. For example, if the current refresh threshold is 4MB, and the current refresh batch already contains a 1MB packet1 with 3MB of remaining storage space, and the MCU acquires a 5MB packet2, then the boundary between the first 3MB and the last 2MB of the packet is used as the dividing point between the current refresh batch and the next refresh batch. At this point, the current refresh batch contains the first 3MB of both packet1 and packet2, and the next refresh batch contains the last 2MB of packet2. It should be noted that although the two parts of packet2 are in different refresh batches, due to the special nature of packets and the continuity of the refresh queue, they are not physically separated; they are only logically divided according to the refresh threshold to prepare for aligned refresh writes to disk. It should be noted that the refresh threshold and packet size are provided to help those skilled in the art better understand the implementation and do not represent the only possible implementation. This embodiment does not impose any special limitations on the packet size or the refresh threshold.

[0081] As can be seen, by logically splitting the acquired packet according to the flush threshold, both the integrity of the packet and the alignment during flushing are guaranteed, thus protecting the data while improving the speed of writing to disk.

[0082] S13: Mark the packet as unsplittered and add the packet to the current batch.

[0083] Specifically, in this embodiment, if the packet size is smaller than the remaining storage size in the current batch, logical splitting of the packet is unnecessary. The acquired packet can be directly marked as unsplit and added to the current batch. For example, if the current batch size threshold is 4MB and the current batch is empty (i.e., has 4MB of storage space), and the acquired packet is 3MB, then the packet size is smaller than the storage space of the current batch and can be directly added to the current batch to await further processing. It should be noted that the batch size threshold and packet size are provided to help those skilled in the art better understand the implementation and do not represent the only possible implementation. This embodiment does not impose any special limitations on the packet size or the batch size threshold.

[0084] As can be seen, by comparing the remaining storage space of the current batch with the size of the packet, if the packet is smaller than the storage space of the current batch, it is directly added to the current batch and waited for the next batch to be flushed, which improves the speed of writing to disk.

[0085] S14: Determine whether the current batch meets the conditions for brushing. If it does, proceed to step S15.

[0086] Specifically, in this embodiment, it is determined whether the current batch meets the brushing conditions. The brushing conditions include the current batch being full, the obtained packet being the first in the block, the obtained packet being the last in the block, and the obtained packet being smaller than the largest packet size in the queue. If the above brushing conditions are met, the current batch can be added to the waiting brushing queue to wait for brushing.

[0087] S15: Add the current batch to the waiting batch queue.

[0088] Specifically, in this embodiment, by filtering the batch of data to be processed, and processing the batch that meets the processing conditions, the data processing can meet the preset requirements, reduce the possibility of garbled characters, and increase the processing speed.

[0089] As can be seen, the packet write alignment method provided in this application specifically involves obtaining the packet sent by the client, determining whether the current flush batch exceeds the flush threshold after adding the packet, and if it does. If it does, the packet is logically split and marked as split, the portion of the packet smaller than the flush threshold is added to the current flush batch, and the remaining portion is added to the next flush batch. If the packet does not exceed the threshold, the packet is marked as unsplit and added to the current flush batch. The method then determines whether the current flush batch meets the flush conditions; if it does, it is added to the waiting flush queue. This achieves packet write alignment processing. By splitting the packet to conform to the flush size, and processing portions that do not conform to the flush size in batches, the write speed to disk can be improved, the possibility of garbled characters can be reduced, and the time spent reading data can be reduced, thus reducing the occurrence of errors.

[0090] Based on the above embodiments, as a preferred embodiment, logically splitting the packet includes:

[0091] The portion of the packet that exceeds the brush threshold is marked as a segmented portion.

[0092] Specifically, in this embodiment, if the packet size exceeds the remaining storage size in the current batch, the packet needs to be logically split. It should be noted that logical splitting is not the same as traditional splitting. Due to the special nature of packets, to ensure no data loss, the portion of the packet exceeding the batch threshold needs to be marked as a split portion. However, the packet remains a single unit, and data loss will not occur due to packet splitting caused by logical splitting.

[0093] As can be seen, by logically splitting the acquired packet according to the flush threshold, both the integrity of the packet and the alignment during flushing are guaranteed, thus protecting the data while improving the speed of writing to disk.

[0094] Based on the above embodiments, as a preferred embodiment, Figure 2 The flowchart for determining whether the current batch meets the conditions for brushing is provided in the embodiments of this application, such as Figure 2 As shown, the condition includes:

[0095] S16: Get the size of the largest packet in the current batch.

[0096] Specifically, in this embodiment, after the MCU obtains a new packet, it needs to update the size of the largest packet among all packets in the current batch. For example, if the current batch contains packet1 with a size of 2M and packet2 with a size of 3M, then the largest packet is packet2. It should be noted that this embodiment does not impose any special limitations on the size or number of packets in the current batch.

[0097] S17: Determine if the packet's sequence number and its offset within the data block are both 0. If both are 0, proceed to step S18.

[0098] Specifically, in this embodiment, it is determined whether the current packet is the first data packet in the first flush batch of the block by judging whether both the sequence number of the packet and the offset in the data block are 0. For example, if the packet is the first packet in the first flush batch, its sequence number and offset in the block are both 0. If the packet is the second packet in the first flush batch, and the previous packet is 64k, then the sequence number of this packet is 1 and the offset is 64k. It should be noted that the examples of sequence numbers and offsets are for the purpose of enabling those skilled in the art to better understand, and do not represent that this is the only implementation method. In this example, there is no special limitation on the sequence number and offset of the packet.

[0099] It is evident that judging only the sequence number or only the offset is insufficient to determine whether the current packet is the first data packet in the first batch of the first flushing in the Block. It is more accurate and specific to determine whether the current packet is the first data packet in the first batch of the first flushing in the Block by judging whether the conditions are met simultaneously.

[0100] S18: Add the packet to the queue waiting for the next flush and replace it with a new flush batch as the next flush batch.

[0101] Specifically, packets with both sequence number and offset of 0 are added to the waiting queue for the next flush and are directly awaited for flushing. Instead of waiting for the next packet to reach the flushing threshold, a new flushing batch is used as the next flushing batch, and new packets are awaited.

[0102] As can be seen, by determining that the packet is the first packet, since there are no other packets in the queue to interfere with its data, the next packet can be flushed directly, saving waiting time.

[0103] S19: Determine if the packet has been split. If it has been split, proceed to step S20; if it has not been split, proceed to step S24.

[0104] Specifically, by determining whether the current data packet is a logically segmented packet, if it has been segmented, the packet is separated at the segmentation point. The portion smaller than the preset downsampling threshold is added to the current downsampling batch, and the remaining portion, i.e., the segmented portion, is added to the next downsampling batch. If it has not been segmented, the packet is directly added to the current downsampling batch.

[0105] S20: Add the portion of the packet smaller than the preset downsweep threshold to the current downsweep batch, and add the already segmented portion to the next downsweep batch.

[0106] If the packet has already been segmented, the packet will be separated from the segmentation point. The portion smaller than the preset downsweep threshold will be added to the current downsweep batch, and the remaining portion, i.e. the segmented portion, will be added to the next downsweep batch.

[0107] S21: Determine if the data length of the packet is less than the size of the largest packet in the next batch. If it is less, proceed to step S22; if it is not less, proceed to step S23.

[0108] S22: Add the packet to the next batch;

[0109] S23: Add the packet to the current batch;

[0110] S24: Add the packet to the current batch.

[0111] In this embodiment, by determining whether the data length of the packet is less than the size of the largest packet in its current batch, if it is less than the size of the packet, it means that the packet is not the same size as the packet in the current batch, and the packet in the current batch can be considered to have been transmitted completely, and the packet can be added to the next batch; otherwise, the packet is added to the current batch.

[0112] It is evident that by filtering the batches and only processing those that meet the processing conditions, the processing can meet the preset requirements, reducing the possibility of garbled characters and increasing the processing speed.

[0113] Based on the above embodiments, as a preferred embodiment, such as... Figure 2 As shown, after determining that the packet's sequence number and its offset within the data block are not both 0, the process also includes:

[0114] S25: Determine if the data length of the packet is less than the size of the largest packet in the next batch. If it is less, proceed to step S18.

[0115] Specifically, in this embodiment, it is determined whether the data length of the current packet is less than the size of the largest packet in the current batch. For example, if the current batch contains packet1 (2MB) and packet2 (3MB), then packet2 is the largest packet. If packet3 (2MB) is received, it is directly added to the waiting batch queue, and a new batch is selected as the next batch. Furthermore, if packet3 is larger than the storage space of the current batch, it is logically split, and the remaining portion is added to the next batch. It should be noted that this embodiment does not impose any special limitations on the size or number of packets in the current batch.

[0116] As can be seen, by determining whether the data length of a packet is less than the size of the largest packet in its current batch, if it is less, it means that the current packet has changed and is different from the largest packet. Packets smaller than the largest packet can be directly added to the current batch, and otherwise added to the current batch.

[0117] Based on the above embodiments, as a preferred embodiment, such as... Figure 2 As shown, after determining that the data length of the packet is not less than the size of the largest packet in its subsequent batch, the process also includes:

[0118] S26: Determine if the packet has been split. If it has been split, proceed to step S27.

[0119] S27: Add the portion of the packet smaller than the preset down-flush threshold to the current down-flush batch, and add the already segmented portion to the next down-flush batch;

[0120] S28: Determine if the packet has been split. If it has not been split, proceed to step S29; if it has been split, proceed to step S30.

[0121] S29: Add the packet to the current batch;

[0122] S30: Add the already segmented portion to the next brush batch.

[0123] Specifically, in this embodiment, it is determined whether the current packet has been segmented. If it has been segmented, the portion of the packet smaller than the preset downsweep threshold is added to the current downsweep batch based on the logical segmentation result and the segmentation point, while the segmented portion is added to the next downsweep batch. If it has not been segmented, the packet is directly added to the current downsweep batch.

[0124] As can be seen, by logically splitting the acquired packet according to the flush threshold, both the integrity of the packet and the alignment during flushing are guaranteed, thus protecting the data while improving the speed of writing to disk.

[0125] Based on the above embodiments, as a preferred embodiment, Figure 2 The flowchart provided in this application embodiment shows the process of determining whether a packet has been split, as follows: Figure 2 As shown, after determining that the packet has not been split, the following steps are also included:

[0126] S31: Determine if the current batch containing the packet exceeds the batch threshold. If it does, proceed to step S32.

[0127] S32: Add the portion of the current batch that exceeds the batch threshold to the waiting batch queue, add the remaining portion to the next batch, and proceed to step S28.

[0128] Specifically, if the current batch containing the packet exceeds the refresh threshold, the portion exceeding the threshold still needs to be logically split and processed in batches. The following inequality can be used to determine whether the current batch exceeds the refresh threshold: batchOffset - alginOffset + batchLen ≥ FLUSH_SIZE, where batchOffset is the offset of the data in the current batch within the file, alginOffset is the aligned offset of the current batch, batchLen is the length of the data in the current batch, and FLUSH_SIZE is the refresh size. This inequality determines whether the current batch containing the packet still exceeds the refresh threshold after the first split. If it does, a second logical split is required. After splitting, the portion smaller than the refresh threshold is added to the current batch, and the remaining portion is added to the next batch.

[0129] As can be seen, by performing a second judgment on the current brush batch, it is possible to prevent the brushing process from being unable to proceed if the current brush batch is still greater than the brushing threshold after the first segmentation. A second logical segmentation process is performed on the current brush batch that still exceeds the brushing threshold.

[0130] Based on the above embodiments, as a preferred embodiment, such as... Figure 2 As shown, after determining that the segmented portion is smaller than the downsweep threshold, the process also includes:

[0131] S33: Determine if the packet is the last packet in the data block it belongs to. If it is the last packet, proceed to step S34; otherwise, proceed to step S35.

[0132] S34: Add the packet to the queue waiting for the next flush and replace it with a new flush batch as the next flush batch.

[0133] S35: Add the packet to the current batch.

[0134] Specifically, by determining whether the packet is the last packet in the block, if it is the last packet, the packet is directly added to the queue waiting for the next flush and a new flush batch is used as the next flush batch. Otherwise, if it is not the last packet, the packet is added to the current flush batch but a new flush batch is not used.

[0135] As can be seen, by determining whether the packet is the last packet in the data block, and then deciding whether to continue waiting for the next packet, unnecessary waiting time is reduced and the flushing efficiency is increased.

[0136] Based on the above embodiments, as a preferred embodiment, Figure 3 The flowchart for the waiting queue processing provided in the embodiments of this application is as follows: Figure 3 As shown, it includes:

[0137] S36: Determine if the packet is not the last packet. If it is not the last packet, proceed to step S37.

[0138] S37: Retrieve the first flush batch from the waiting flush queue, and determine if the last packet in the first flush batch is the last packet in its data block. If it is the last packet, proceed to step S38.

[0139] S38: Determine if there is only one packet in the first batch. If there is only one, proceed to step S39; otherwise, proceed to step S40.

[0140] S39: Add the first batch to the confirmation set and wait for client confirmation, return confirmation to the client, remove the first batch from the waiting batch queue, and return to step S36.

[0141] S40: Add the first batch of data to the confirmation set and wait for client confirmation. After the first batch of data is processed, remove the first batch of data from the waiting queue and return to step S36.

[0142] If it is not the last one, proceed to step S40.

[0143] Specifically, in this embodiment, when the last packet in the current batch is the last packet in the block and there is only one packet in the batch, the client has received confirmation information for all data packets, indicating that the data has been written completely, and the packet can be directly returned as confirmation. If the packet is not the last packet, the first batch is taken from the current batch queue. It is then checked whether the last packet in this batch is the last packet in its block. If it is the last, it is further checked whether there is only one packet in the first batch. If there is only one, the first batch is added to the confirmation set for client confirmation, and confirmation is returned to the client. The first batch is removed from the waiting batch queue, and the process returns to step S36. Conversely, if the first batch is not the last, the process continues with step S37.

[0144] As can be seen, when it is confirmed that the last packet in the current batch is the last packet in the block, and there is only one packet in the batch, the client has received confirmation information for all data packets, indicating that the data has been written and the packet can be directly returned as confirmation.

[0145] Based on the above embodiments, as a preferred embodiment, such as... Figure 3 As shown, after determining that the packet is the last packet, the following steps are also taken:

[0146] Determine if the queue waiting to be refreshed is not empty.

[0147] If it is not empty, proceed to step S37.

[0148] After determining that the packet is the last packet, it is necessary to check if the current queue waiting for flushing is empty. If it is not empty, it means there are still packets to be flushed, and the process returns to S37 for further judgment. If it is empty, the flushing is complete, and there are no new packets to flush. It should be noted that this embodiment uses asynchronous flushing. After the flushing is completed, confirmation is achieved through a callback. When the flushing completion callback is received, the batch is retrieved from the confirmation set using the batchOffset passed in the callback, and all packets in the batch are confirmed. There is a special case here: because we reorganized the packets in the above process, some packets were split and added to two batches. Therefore, confirmation can only be performed after receiving the callbacks for two batches. The number of times a packet should be confirmed (shouldAck) records the number of times the packet needs to receive callbacks. Only when the number is 0 will the packet be acked to the client.

[0149] Figure 4 The flowchart for packet processing, downloading, and confirmation provided in the embodiments of this application is as follows: Figure 4 As shown, client ① sends a packet to the DataNode. The DataNode contains a BlockReceiver and a PacketReceiver. PacketReceiver ② receives and parses the packet, ③ reorganizes the packet and adds it to the current batch. Here, ③ corresponds to the above... Figure 1 , Figure 2 In the steps of the content, ④ add to the flush queue, corresponding to Figure 3 The steps are as follows: after organizing the flush queue, ⑤ align the flush data, ⑥ add it to the confirmation queue, ⑦ after the flush is completed, a callback is made, and in the confirmation queue, ⑧ use the responder to reply with an Ack (acknowledgment) to the client, and ⑨ the client receiving an Ack indicates that the packet writing is complete.

[0150] The packet write alignment method has been described in detail in the above embodiments. This application also provides embodiments corresponding to the packet write alignment system. It should be noted that this application describes the system-related embodiments from two perspectives: one based on functional modules and the other based on hardware.

[0151] From the perspective of functional modules, this embodiment provides a packet write alignment system. Figure 5 This is a structural diagram of the packet write alignment system provided in the embodiments of this application, as shown below. Figure 5As shown, the system includes:

[0152] Module 10 is used to retrieve packets sent by the client.

[0153] The first judgment module 11 is used to determine whether the current batch exceeds the brush threshold after the packet is added to the current batch.

[0154] The first segmentation module 12 is used to logically segment the packet if the threshold is exceeded and mark it as already segmented, add the portion of the packet smaller than the down-flush threshold to the current down-flush batch, and add the remaining portion to the next down-flush batch;

[0155] The second segmentation module 13 is used to mark the packet as unsegmented if the number of packets does not exceed the limit, and to add the packet to the current batch.

[0156] The second judgment module 14 is used to determine whether the current batch meets the conditions for brushing.

[0157] Add module 15 to add the current batch to the waiting batch queue.

[0158] Since the embodiments of the system part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the system part, and they will not be repeated here.

[0159] The packet writing alignment system provided in this embodiment corresponds to the method described above, and therefore has the same beneficial effects as the method described above.

[0160] From a hardware perspective Figure 6 This is a structural diagram of a packet write alignment device provided in another embodiment of this application, as shown below. Figure 6 As shown, it includes a memory 20 for storing computer programs;

[0161] Processor 21 is used to implement the packet write alignment method described above when executing a computer program.

[0162] The packet write alignment device provided in this embodiment may include, but is not limited to, servers.

[0163] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0164] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of the packet write alignment method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, data related to the packet write alignment method.

[0165] In some embodiments, the packet writing alignment device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0166] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on the packet write alignment device and may include more or fewer components than illustrated.

[0167] The packet write alignment device provided in this application includes a memory and a processor. When the processor executes a program stored in the memory, it can implement the method described above.

[0168] Finally, this application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above method embodiments.

[0169] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0170] The foregoing has provided a detailed description of a packet alignment method, system, apparatus, and medium provided in this application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0171] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A packet write alignment method, characterized in that, include: Retrieve the packet sent by the client; Determine whether the current batch exceeds the refresh threshold after the packet is added to the current batch. If the threshold is exceeded, the packet is logically segmented and marked as segmented. The portion of the packet smaller than the threshold is added to the current batch, and the remaining portion is added to the next batch. The logical segmentation of the packet includes marking the portion of the packet exceeding the threshold as segmented. If the number of packets does not exceed the limit, the packet is marked as unsplitled and added to the current batch. Determine whether the current batch meets the conditions for being refreshed; If the conditions are met, the current batch is added to the waiting batch queue. The step of determining whether the current batch meets the conditions for brushing includes: Get the largest packet size in the current batch. Determine whether the sequence number of the packet and its offset in the data block are both 0; If both the sequence number of the packet and its offset in the data block are 0, then the packet is added to the queue waiting for flushing, and a new flushing batch is used as the next flushing batch. Determine whether the packet has been split; If the packet has already been segmented, the portion of the packet smaller than the downsweep threshold is added to the current downsweep batch, and the segmented portion is added to the next downsweep batch. It is then determined whether the data length of the packet is less than the largest packet size in the downsweep batch. If it is less, the packet is added to the next downsweep batch; if it is not less, the packet is added to the current downsweep batch. If it has not been split, then the packet is added to the current batch. If the sequence number of the packet and the offset in the data block are not both 0, then it is determined whether the data length of the packet is less than the size of the largest packet in the next flush batch; if it is less, then proceed to the step of adding the packet to the waiting flush queue and replacing it with a new flush batch as the next flush batch.

2. The packet write alignment method according to claim 1, characterized in that, After determining that the data length of the packet is not less than the size of the largest packet in the next batch, the method further includes: Determine whether the packet has been split; If the packet has already been segmented, the portion of the packet smaller than the down-flush threshold is added to the waiting down-flush queue, and the segmented portion is added to the next down-flush batch. Determine whether the packet has been split; If it has not been split, then the packet is added to the current batch. If it has already been segmented, then the segmented portion is added to the next brush batch.

3. The packet write alignment method according to claim 2, characterized in that, After determining that the packet has not been split, the method further includes: Determine whether the current batch containing the packet exceeds the batch threshold; If the threshold is exceeded, the portion of the current batch that exceeds the threshold is added to the waiting batch queue, the remaining portion is added to the next batch, and the process proceeds to the step of determining whether the packet has been split.

4. The packet write alignment method according to claim 3, characterized in that, After determining that the current batch containing the packet is less than the brush threshold, the process further includes: Determine whether the packet is the last packet in the data block it belongs to; If it is the last packet, then the packet is added to the waiting queue for the next flush, and a new flush batch is used as the next flush batch, and the process proceeds to the step of determining whether the packet has been split. If it is not the last packet, then the packet is added to the current batch.

5. The packet write alignment method according to claim 1, characterized in that, After adding the current batch to the waiting batch queue, the process further includes: Determine whether the packet is not the last packet. If it is not the last packet, then the first batch of data is obtained from the waiting batch of data, and it is determined whether the last packet in the first batch of data is the last packet in the data block. If it is the last one, then determine whether there is only one packet in the first batch of downloads; If there is only one, add the first batch of data to the confirmation set and wait for the client to confirm. Return the confirmation to the client, remove the first batch of data from the waiting queue, and return to the step of determining whether the packet is not the last packet. If there is more than one, the first batch is added to the confirmation set and awaits client confirmation. After the first batch is completed, the process proceeds to remove the first batch from the waiting batch queue and returns to the step of determining whether the packet is not the last packet. If it is not the last one, then proceed to the step of adding the first batch of downloads to the confirmation set and waiting for the client to confirm. After the first batch of downloads is completed, proceed to the step of removing the first batch of downloads from the waiting queue.

6. The packet write alignment method according to claim 5, characterized in that, After determining that the packet is the last packet, the process further includes: Determine whether the queue waiting for the next flush is not empty; If it is not empty, proceed to the step of obtaining the first batch of data from the waiting batch and determining whether the last packet in the first batch is the last packet in the data block.

7. A packet write alignment system, characterized in that, include: The acquisition module is used to retrieve packets sent by the client. The first judgment module is used to determine whether the current batch exceeds the brush threshold after the packet is added to the current batch. The first segmentation module is used to, if the threshold is exceeded, logically segment the packet and mark it as already segmented, add the portion of the packet smaller than the downsweep threshold to the current downsweep batch, and add the remaining portion to the next downsweep batch; wherein, logically segmenting the packet includes: marking the portion of the packet exceeding the downsweep threshold as already segmented; The second segmentation module is used to mark the packet as unsegmented if the number of segments does not exceed a certain threshold, and to add the packet to the current batch. The second judgment module is used to determine whether the current batch meets the brushing conditions; The add module is used to add the current batch to the waiting batch queue; The step of determining whether the current batch meets the conditions for brushing includes: Get the largest packet size in the current batch. Determine whether the sequence number of the packet and its offset in the data block are both 0; If both the sequence number of the packet and its offset in the data block are 0, then the packet is added to the queue waiting for flushing, and a new flushing batch is used as the next flushing batch. Determine whether the packet has been split; If the packet has already been segmented, the portion of the packet smaller than the downsweep threshold is added to the current downsweep batch, and the segmented portion is added to the next downsweep batch. It is then determined whether the data length of the packet is less than the largest packet size in the downsweep batch. If it is less, the packet is added to the next downsweep batch; if it is not less, the packet is added to the current downsweep batch. If it has not been split, then the packet is added to the current batch. If the sequence number of the packet and the offset in the data block are not both 0, then it is determined whether the data length of the packet is less than the size of the largest packet in the next flush batch; if it is less, then proceed to the step of adding the packet to the waiting flush queue and replacing it with a new flush batch as the next flush batch.

8. A packet write alignment device, characterized in that, Includes memory used to store computer programs; A processor, configured to implement the packet write alignment method as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the packet write alignment method as described in any one of claims 1 to 6.

Images