Streaming data sending method and device and streaming data integrity checking method

Abstract

This invention discloses a streaming data transmission method, apparatus, and streaming data integrity verification method. The streaming data transmission method includes: obtaining verification file information corresponding to a current first file to be verified; reading a verification data block of a preset size from the current first file to be verified according to the verification file read offset; inserting the verification file name, verification file read offset, all or part of the first digest calculation result, and the verification data block into a business streaming data packet corresponding to the business data; and sending the inserted business streaming data packet to the data receiver. The verification file has a natural integrity verification method. By performing digest calculation on the verification file, a deterministic result is obtained. Modifying any data in the verification file will lead to a change in the digest calculation result. Based on the mechanism of sending verification file information, problems such as transmission omissions and failures at the data sender can be eliminated, thereby improving the accuracy of data stream integrity verification.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a streaming data transmission method, apparatus, and streaming data integrity verification method. Background Technology

[0002] In today's big data era, an increasing amount of data needs to be transmitted in real time between various data platforms via the network. To reduce the latency between data generation and processing, a common approach is to send the generated data immediately, rather than accumulating it into a certain volume before sending it in batches. The former is usually called real-time transmission, while the latter is called batch transmission.

[0003] In real-time transmission scenarios, when the data sender and receiver verify the amount of data, they often use a counting method. The sender increments a counter by one for each successfully sent data item, and the receiver maintains a similar counter during data reception and processing. By comparing these counters, both parties determine whether any data has been lost during real-time transmission.

[0004] A better approach is to count based on timestamps. For example, when the data sender transmits data, a current timestamp is added to the data. During data reception and processing, the data receiver uses the added timestamps to count the amount of data belonging to a specific time period. By comparing the counts at different monitoring points within the same time period, it's possible to determine which monitoring points experienced data loss and during which time periods the loss occurred. In summary, existing technical solutions include: 1. Using a counting method to collect data volume at each monitoring point and comparing the data volume to determine data stream integrity. 2. In addition to the counting method, adding a timestamp factor to improve the accuracy of data stream integrity verification.

[0005] However, the above data stream integrity verification has the following problems:

[0006] 1. Using a counting method is prone to errors. Since the real-time data stream is uninterrupted, each monitoring point needs to define a range for data volume comparison, such as the data volume within one hour. However, because time cannot be precisely synchronized, there will inevitably be errors in the data volume.

[0007] 2. Combining timestamp integrity verification schemes can eliminate errors caused by inconsistent scope definitions. However, it can only determine the consistency of statistical data from each monitoring node, and cannot identify problems originating from the initial monitoring point, such as missing data actually sent by the initial monitoring point. Summary of the Invention

[0008] In view of the above problems, embodiments of the present invention are proposed to provide a streaming data transmission method, apparatus and streaming data integrity verification method that overcome the above problems or at least partially solve the above problems.

[0009] According to one aspect of the present invention, a streaming data transmission method is provided, comprising:

[0010] Obtain the verification file information corresponding to the first file to be verified, including: verification file name, verification file read offset, and first digest calculation result;

[0011] Read a pre-sized block of verification data from the current first file to be verified based on the offset of the verification file;

[0012] Insert the verification file name, verification file read offset, all or part of the first digest calculation result, and verification data block into the business streaming data packet corresponding to the business data, and send the inserted business streaming data packet to the data receiver.

[0013] According to another aspect of the present invention, a streaming data integrity verification method is provided, comprising:

[0014] Receive business streaming data packets sent by the data sender;

[0015] Extract the verification file name, verification file read offset, verification data block, and first digest calculation result from at least one business streaming data packet;

[0016] Write the verification data block to the second file to be verified corresponding to the verification file name based on the offset read from the verification file.

[0017] If no business streaming data packet is received within the preset time period, a digest calculation is performed on the second file to be verified to obtain the second digest calculation result;

[0018] Match the results of the second summary calculation with the results of the first summary calculation;

[0019] Based on the matching results, determine the data integrity verification result.

[0020] According to another aspect of the present invention, a streaming data transmission apparatus is provided, comprising:

[0021] The acquisition module is adapted to acquire the verification file information corresponding to the current first file to be verified, wherein the verification file information includes: verification file name, verification file read offset, and first digest calculation result;

[0022] The reading module is adapted to read a preset size of verification data block from the current first file to be verified according to the verification file reading offset;

[0023] The insertion module is suitable for inserting the verification file name, the verification file read offset, all or part of the first digest calculation result, and the verification data block into the business streaming data packet corresponding to the business data.

[0024] The sending module is suitable for sending the inserted business streaming data packets to the data receiver.

[0025] According to another aspect of the present invention, a streaming data integrity verification device is provided, comprising:

[0026] The receiving module is suitable for receiving business streaming data packets sent by the data sender;

[0027] The extraction module is suitable for extracting the verification file name, verification file read offset, verification data block, and first digest calculation result from at least one business streaming data packet.

[0028] The writing module is adapted to write the verification data block to the second file to be verified corresponding to the verification file name according to the offset read from the verification file.

[0029] The calculation module is adapted to perform a digest calculation on the second file to be verified if no business streaming data packet is received within a preset time period, and obtain the second digest calculation result;

[0030] A matching module, adapted to match the second summary calculation result with the first summary calculation result;

[0031] The determination module is suitable for determining the streaming data integrity verification result based on the matching result.

[0032] According to another aspect of the present invention, a computing device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0033] The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the above-described streaming data transmission method or streaming data integrity verification method.

[0034] According to another aspect of the present invention, a computer storage medium is provided, wherein at least one executable instruction is stored in the storage medium, the executable instruction causing a processor to perform operations corresponding to the above-described streaming data transmission method or streaming data integrity verification method.

[0035] The solution provided by this invention has a natural integrity verification method for the verification file. A deterministic result is obtained by performing a digest calculation on the verification file. Modifying any data in the verification file will cause the digest calculation result to change. Therefore, based on the mechanism of the data sender sending verification file information (verification file name, verification file read offset, and all or part of the first digest calculation result), problems such as sending omissions or failures can be eliminated, thereby improving the accuracy of data stream integrity verification.

[0036] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more obvious and understandable, specific implementation methods of the embodiments of the present invention are described below. Attached Figure Description

[0037] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0038] Figure 1A A flowchart of the streaming data transmission method provided in an embodiment of the present invention is shown;

[0039] Figure 1B A schematic diagram of a service streaming data packet carrying a specified number of bits of the first digest calculation result;

[0040] Figure 1C This is a flowchart illustrating the process of reading a verification data block based on the offset from the verification file.

[0041] Figure 2 A flowchart of the streaming data integrity verification method provided in an embodiment of the present invention is shown;

[0042] Figure 3 A schematic diagram of the structure of the streaming data transmission device provided in an embodiment of the present invention is shown;

[0043] Figure 4 A schematic diagram of the structure of the streaming data integrity verification device provided in an embodiment of the present invention is shown;

[0044] Figure 5 A schematic diagram of the structure of a computing device provided in an embodiment of the present invention is shown. Detailed Implementation

[0045] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0046] Figure 1A A flowchart of a streaming data transmission method provided in an embodiment of the present invention is shown. In this embodiment, the execution entity is the data sender, such as… Figure 1A As shown, the method includes the following steps:

[0047] Step S101: Obtain the verification file information corresponding to the current first file to be verified, wherein the verification file information includes: verification file name, verification file read offset, and first digest calculation result.

[0048] To accurately verify data integrity, the data sender needs to process the business streaming data according to the method provided in this embodiment before sending it. Specifically, this involves obtaining the verification file information corresponding to the current first file to be verified. This verification file information includes details related to the current first file to be verified, such as the verification file name, the verification file read offset, and the first digest calculation result. In this step, the first file to be verified is relative to the data sender; the current first file to be verified is the first file that needs to be verified. In practical applications, there may be multiple first files to be verified.

[0049] The verification file read offset represents the offset of the verification data block that has already been read in the first file to be verified. It is used to indicate from which position to start reading data when reading data in the verification file again.

[0050] For setting the verification file name, it is recommended to use a fixed length and a numeric format.

[0051] The first digest calculation result is obtained by performing a digest calculation on the current first file to be verified. For example, any digest calculation algorithm such as MD2, MD4, MD5, SHA-1, or SHA-2 can be used to perform a digest calculation on the current first file to be verified to obtain the corresponding first digest calculation result. The first digest calculation result has a fixed length, such as 32 bits or 64 bits.

[0052] Step S102: Read a preset size of verification data block from the current first file to be verified according to the verification file reading offset.

[0053] After obtaining the read offset of the current first file to be verified, a pre-defined size of verification data block can be read from the current first file to be verified based on the read offset. For example, starting from the position corresponding to the read offset in the current first file to be verified, a pre-defined size of verification data block is read. The size of the read verification data block is fixed and pre-set; the pre-defined size can be understood as the data block length of the verification data block. To avoid the inability to read the complete data block, it is agreed that the size of the current first file to be verified is an integer multiple of the data block length.

[0054] Step S103: Insert the verification file name, verification file read offset, all or part of the first digest calculation result, and verification data block into the business streaming data packet corresponding to the business data, and send the inserted business streaming data packet to the data receiver.

[0055] After reading a pre-defined verification data block, the system inserts the verification file name, verification file read offset, all or part of the first digest calculation result, and the verification data block of the current first file to be verified into the corresponding business streaming data packet of the business data, according to a predetermined format. For example, the insertion positions of the verification file name, verification file read offset, all or part of the first digest calculation result, and the verification data block are predetermined. Following this, the system inserts the verification file name, verification file read offset, all or part of the first digest calculation result, and the verification data block of the current first file to be verified into the corresponding business streaming data packet of the business data. After the insertion is completed, the inserted business streaming data packet is sent to the data receiver.

[0056] The business streaming data packet is a data packet obtained after the business data is segmented. The information such as the verification file name, the verification file read offset, all or part of the first digest calculation result, and the verification data block are sent to the data receiver along with the business streaming data packet. The data receiver does not need to verify whether the business data is complete, but only needs to verify whether the received verification data block is complete. The data contained in the first file to be verified is much smaller than the business data, thereby improving the efficiency of streaming data integrity verification.

[0057] This embodiment uses the transmission of a single service streaming data packet as an example. The information inserted into the service streaming data packet corresponding to the service data varies depending on the circumstances. For example, the verification file name, verification file read offset, the entire first digest calculation result, and the verification data block can be inserted into each service streaming data packet corresponding to the service data. This method ensures the accuracy of integrity verification. Another example is inserting the verification file name, verification file read offset, a portion of the first digest calculation result, and the verification data block into each service streaming data packet corresponding to the service data. This method reduces the amount of information carried in the service streaming data packet. Yet another example is inserting the verification file name, verification file read offset, a portion of the first digest calculation result, and the verification data block into a portion of the service streaming data packet, while for other portions of the service streaming data packet, only the verification file name, verification file read offset, and verification data block are inserted.

[0058] In an optional embodiment of the present invention, for the case of inserting a portion of the first digest calculation result into the service streaming data packet, inserting the verification file name, the verification file read offset, a portion of the first digest calculation result, and the verification data block into the service streaming data packet corresponding to the service data further includes:

[0059] The offset read from the verification file is used to take the remainder of the length of the first digest calculation result, and the value of a specified bit in the first digest calculation result is determined based on the first remainder value.

[0060] Insert the verification file name, verification file read offset, the value of a specified bit of the first digest calculation result, and the verification data block into the business streaming data packet corresponding to the business data.

[0061] Specifically, the offset of the verification file read is moduloed by the length of the first digest calculation result to obtain the remainder, referred to as the first remainder value. The value of a specified bit in the first digest calculation result is then obtained according to this first remainder value. For example, in this embodiment, after performing the remainder operation, the remainder is 5, so the value of the 5th bit of the first digest calculation result needs to be obtained. After determining the value of the specified bit in the first digest calculation result, the verification file name, verification file read offset, and the value of the specified bit in the first digest calculation result are inserted into the service streaming data packet. This transmission only includes the value of the specified bit in the first digest calculation result. The purpose of this is to reduce the size of the verification file information carried in the service streaming data packet. In this case, each service streaming data packet can contain only the value of the specified bit in the first digest calculation result.

[0062] To further reduce the repeated transmission of the first digest calculation result, this embodiment can insert a specified number of bits of the first digest calculation result into some service streaming data packets, while not inserting the specified number of bits of the first digest calculation result into other service streaming data packets. The service streaming data packets corresponding to the service data include: a first service streaming data packet and a second service streaming data packet; the verification file name, verification file read offset, part of the first digest calculation result, and verification data block are inserted into the first service streaming data packet corresponding to the service data; the verification file name, verification file read offset, and verification data block are inserted into the second service streaming data packet corresponding to the service data.

[0063] Specifically, the following methods can be used to determine which service streaming data packets are the first service streaming data packets and which are the second service streaming data packets: Check if the length of the verification file name is greater than the length of the first digest calculation result. If the length of the verification file name is greater than the length of the first digest calculation result, then the second remainder value is calculated using formula (1):

[0064] Second remainder value = (check file read offset / check data block length) % Check file name length formula (1);

[0065] If the second remainder value equals 0, the service streaming data packet corresponding to that moment is identified as the first service streaming data packet. Starting from this service streaming data packet, multiple consecutive service streaming data packets are identified as first service streaming data packets. The number of consecutive service streaming data packets corresponds to the length of the first digest calculation result. For example, if the length of the first digest calculation result is 32 bits, then starting from the first identified first service streaming data packet, 32 consecutive service streaming data packets are identified as first service streaming data packets.

[0066] Inserting the verification file name, verification file read offset, part of the first digest calculation result, and verification data block into the first business streaming data packet corresponding to the business data can be further achieved through the following method:

[0067] If the second remainder value is equal to 0, then multiple consecutive first service streaming data packets corresponding to the service data are determined. For each first service streaming data packet, the length of the first digest calculation result is moduloed by the verification file read offset. The value of the specified bit of the first digest calculation result is determined according to the third remainder value. The verification file name, verification file read offset, the value of the specified bit of the first digest calculation result, and the verification data block are respectively inserted into the first service streaming data packet.

[0068] For each specific first service stream data packet, a value of a specified bit in the first digest calculation result will be inserted. The specific bit value to be inserted is determined by taking the remainder of the first digest calculation result length from the offset read from the check file, based on the third remainder value, which will not be elaborated here.

[0069] If the second remainder value is not equal to 0, it indicates that the insertion of the specified bit value of the first digest result is not satisfied. In this case, the verification file name, verification file read offset, and verification data block can be inserted into the second business streaming data packet corresponding to the business data. The operation of "taking the remainder of the first digest result length using the verification file read offset" will not be performed at this time.

[0070] Optionally, the first service streaming data packet into which the specified bits of the first digest calculation result are inserted may not be consecutive. Which service streaming data packets can be used as the first service streaming data packet can be determined based on whether the second remainder value is equal to 0. Because streaming data transmission is large, it is possible to guarantee the transmission of a complete first digest calculation result.

[0071] Figure 1B This is a schematic diagram of a service streaming data packet carrying a specified number of bits of the first digest calculation result. Figure 1B The text only illustrates that a specified number of bits containing the first digest calculation result can be inserted into a business streaming data packet, without indicating any other information. Figure 1B The N in the N business streaming data packets shown corresponds to the length N of the verification file name, which facilitates the subsequent assembly and verification of the first digest calculation results and also improves flexibility.

[0072] In an optional embodiment of the present invention, before obtaining the verification file information corresponding to the current first file to be verified, the method further includes:

[0073] Based on the pre-configured verification file directory, obtain at least one first file to be verified;

[0074] Perform digest calculation on at least one first document to be verified to obtain the first digest calculation result of the first document to be verified.

[0075] Store the verification file name, first digest calculation result, and verification file read offset corresponding to at least one first file to be verified into a circular linked list;

[0076] Obtaining the verification file information corresponding to the current first file to be verified further includes:

[0077] Retrieve the verification file information of the first file to be verified from the circular linked list.

[0078] Optionally, reading a preset-size verification data block from the current first file to be verified according to the verification file read offset further includes:

[0079] Determine whether the file read offset is the offset from the end of the file;

[0080] If so, initialize the verification file read offset and retrieve the verification file information of the next first file to be verified from the circular linked list;

[0081] If not, then read a preset size of verification data block from the current first file to be verified based on the verification file read offset.

[0082] Specifically, Figure 1C This is a flowchart illustrating the process of reading a check data block based on the offset from the check file, as shown below. Figure 1C As shown in (1), during initialization, the configured verification file directory is scanned to obtain all first files to be verified. A digest calculation is performed on each first file to be verified, and the verification file name, the first digest calculation result, and the verification file read offset are stored in a circular linked list.

[0083] (2) Retrieve the verification file information of the current first file to be verified from the circular linked list that stores the verification file information. Each first file to be verified can be called a linked list node. For example, read the verification file name, the verification file read offset, and the first digest calculation result corresponding to the current first file to be verified. It is recommended that the verification file name be of fixed length and in numeric format.

[0084] (3) Open the file according to the file name and the file read offset, and read a fixed-size verification data block. The size of the current first file to be verified is agreed to be an integer multiple of the data block length to avoid being unable to read the complete verification data block. Therefore, before reading the verification data block, it is necessary to determine whether the file read offset is the file end offset. If the file read offset has reached the file end offset, initialize the file read offset to 0, point the current verification file information to the next first file to be verified node in the circular linked list, and execute step (2); otherwise, read a preset-size verification data block from the current first file to be verified according to the file read offset.

[0085] After the insertion is completed, the method also includes: updating the check file read offset according to the size of the check data block, for example, increasing the check file read offset by the data block length, and then storing it in a circular linked list.

[0086] The solution provided by this invention has a natural integrity verification method for the verification file. A deterministic result is obtained by performing a digest calculation on the verification file. Modifying any data in the verification file will cause the digest calculation result to change. Therefore, based on the mechanism of the data sender sending verification file information (verification file name, verification file read offset, and all or part of the first digest calculation result), problems such as sending omissions or failures can be eliminated, thereby improving the accuracy of data stream integrity verification.

[0087] Figure 2 A flowchart of a streaming data integrity verification method according to another embodiment of the present invention is shown. Figure 2 As shown, the method includes the following steps:

[0088] Step S201: Receive the service streaming data packet sent by the data sender.

[0089] In this embodiment, the execution entity is the data receiver, which receives the service streaming data packets sent by the data provider.

[0090] Step S202: Extract the verification file name, verification file read offset, verification data block, and first digest calculation result from at least one business streaming data packet.

[0091] Since the insertion position of the information inserted into the business streaming data packet is predetermined, after receiving the business streaming data packet, the data receiver can extract the verification file name, verification file read offset, verification data block, and first digest calculation result from at least one business streaming data packet.

[0092] The information inserted into the corresponding business streaming data packets differs under different circumstances. Therefore, the information extracted by the data receiver from each business streaming data packet is different. For example, it can extract the entirety of the verification file name, verification file read offset, verification data block, and first digest calculation result from each business streaming data packet; or, for another example, it can extract a portion of the verification file name, verification file read offset, first digest calculation result, and verification data block from each business streaming data packet; or, for yet another example, it can extract the verification file name, verification file read offset, first digest calculation result, and verification data block from a portion of the business streaming data packets, and extract the verification file name, verification file read offset, and verification data block from other portions of the business streaming data packets.

[0093] Therefore, the check file name, check file read offset, check data block, and first digest calculation result can be extracted from at least one business streaming data packet.

[0094] Step S203: Write the verification data block to the second file to be verified corresponding to the verification file name according to the offset read from the verification file.

[0095] After extracting the verification file read offset and verification data block, the verification data block is written to the second file to be verified corresponding to the verification file name according to the verification file read offset. The second file to be verified here is relative to the data receiver and is the file generated after the data receiver writes the received verification data block.

[0096] Step S204: If no service streaming data packet is received within the preset time period, a digest calculation is performed on the second file to be verified to obtain the second digest calculation result.

[0097] Step S205: Match the second summary calculation result with the first summary calculation result.

[0098] Step S206: Determine the data integrity verification result based on the matching result.

[0099] To reduce the probability of false alarms in integrity verification, integrity verification of the second file to be verified is only performed when the data receiver receives business streaming data packets for more than 5 minutes. For example, the statistical duration of not receiving business streaming data packets sent by the data sender is counted. If the statistical duration is greater than or equal to the preset duration (5 minutes), a digest calculation is performed on the second file to be verified to obtain the second digest calculation result.

[0100] Integrity verification requires using the same digest calculation method as the data sender when calculating the first digest. For example, if the data sender uses the MD5 algorithm to calculate the first digest, the data receiver should also use the MD5 algorithm to calculate the second digest. The digest calculation is performed on the second file to be verified, and the second digest result is compared with the first digest result. If the second digest result is the same as the first digest result, it can be determined that the business data received by the data receiver is the same as the business data sent by the data sender, and the data is complete. If the second digest result is different from the first digest result, it can be determined that business data has been lost.

[0101] In an optional embodiment of the present invention, extracting the check file name, check file read offset, check data block, and first digest calculation result from at least one business streaming data packet further includes:

[0102] If the length of the verification file name is greater than the length of the first digest calculation result, then the fourth remainder value is calculated using formula (2):

[0103] Fourth remainder value = (check file read offset / check data block length)% Check file name length formula (2);

[0104] If the fourth remainder value is equal to 0, then extract the check file name, check file read offset, the value of a specified bit of the first digest calculation result, and the check data block from the business streaming data packet.

[0105] If the fourth remainder value is not equal to 0, then extract the verification file name, verification file read offset, and verification data block from the business streaming data packet;

[0106] If the length of the verification file name is less than or equal to the length of the first digest calculation result, then the verification file name, verification file read offset, verification data block, and first digest calculation result are extracted from at least one business streaming data packet.

[0107] Specifically, when sending service streaming data packets, in order to reduce the size of the information carried in the service streaming data packets and to further reduce the repeated transmission of the first digest calculation result, the data sender inserts a specified bit of the first digest calculation result into some service streaming data packets. Therefore, in order to accurately extract the information inserted into the service streaming data packets, it is necessary to determine whether the service streaming data packets contain the specified bit of the first digest calculation result. First, it is determined whether the length of the verification file name is greater than the length of the first digest calculation result. If the length of the verification file name is greater than the length of the first digest calculation result, then the fourth remainder value is calculated using formula (2):

[0108] Fourth remainder value = (check file read offset / check data block length)% Check file name length formula (2);

[0109] If the fourth remainder value is equal to 0, it is determined that the specified bit value of the first digest calculation result exists in the business streaming data packet, and the verification file name, verification file read offset, the specified bit value of the first digest calculation result, and the verification data block can be extracted from the business streaming data packet; if the fourth remainder value is not equal to 0, it is determined that the specified bit value of the first digest calculation result does not exist in the business streaming data packet, and the verification file name, verification file read offset, and verification data block can be extracted from the business streaming data packet.

[0110] If it is determined that a value exists for a specified bit in the first digest calculation result, the method further includes: taking the remainder of the length of the first digest calculation result using the offset read from the verification file, and storing the value of the specified bit in the first digest calculation result into a specified location according to the fifth remainder value, so as to assemble the first digest calculation result.

[0111] When the algorithm used for digest calculation is determined, the length of the digest result is also determined. Therefore, the data receiver can perform a modulo operation on the length of the first digest result using the offset from the verification file read, obtaining the corresponding remainder, i.e., the fifth remainder value. For example, in this embodiment, after performing the modulo operation, the remainder is 5. Therefore, the value of a specified bit in the first digest result needs to be stored in the 5th bit to assemble the first digest result. It should be noted that the service streaming data packets are continuously sent and received; therefore, it is certain that the first digest result can be assembled.

[0112] If the specified bits of the first digest calculation result are received subsequently, they can be compared with the corresponding specified bits of the assembled first digest calculation result. If they match after three matches, the corresponding first digest calculation result can be determined.

[0113] Optionally, there may be a situation where each business streaming data packet has a specified bit of the first digest calculation result inserted. In this case, it is possible to determine whether the length of the verification file name is greater than the length of the first digest calculation result. If it is greater, the length of the first digest calculation result can be moduloed by the verification file read offset. The specified bit of the first digest calculation result can be stored in the specified position according to the fifth remainder value to assemble the first digest calculation result.

[0114] In an optional embodiment of the present invention, writing the verification data block to the second file to be verified corresponding to the verification file name according to the verification file read offset further includes:

[0115] If the first summary calculation result is not obtained by assembly, the verification data block is written to the second file to be verified corresponding to the verification file name according to the verification file read offset;

[0116] If the first digest calculation result is obtained by assembly, the second file to be verified is renamed according to the verification file name and the first digest calculation result to generate a new verification file name. The verification data block is written to the second file to be verified corresponding to the new verification file name according to the verification file read offset.

[0117] Specifically, when the first summary calculation result is not assembled, a second file to be verified is created according to the verification file name or the second file to be verified corresponding to the verification file name is opened, and the verification data block is written to the second file to be verified corresponding to the verification file name according to the verification file read offset.

[0118] If the first digest calculation result is obtained by assembly, the second file to be verified is renamed according to the verification file name and the first digest calculation result, and a new verification file name is generated in the format of "verification file name - first digest calculation result". The verification data block is written to the second file to be verified corresponding to the new verification file name according to the verification file read offset.

[0119] The solution provided by this invention, upon receiving a service streaming data packet from a data sender, extracts the first digest calculation result and writes the data in the verification data block into a second file to be verified. During integrity verification, a second digest calculation result is generated for the second file to be verified, and the second digest calculation result is matched with the first digest calculation result. The data integrity verification result is determined based on the matching result, thereby accurately determining whether the received service data has been lost. Furthermore, by transmitting verification file information, the trust between end-to-end nodes is increased. The verification file can be transmitted to each end-to-end node, and each node can perform integrity checks based on the verification file information.

[0120] Figure 3 A schematic diagram of the structure of a streaming data transmission device provided in an embodiment of the present invention is shown. Figure 3 As shown, the device includes: an acquisition module 301, a reading module 302, an insertion module 303, and a sending module 304.

[0121] The acquisition module 301 is adapted to acquire the verification file information corresponding to the current first file to be verified, wherein the verification file information includes: verification file name, verification file read offset, and first digest calculation result;

[0122] The reading module 302 is adapted to read a preset size of verification data block from the current first file to be verified according to the verification file reading offset;

[0123] The insertion module 303 is adapted to insert the verification file name, the verification file read offset, all or part of the first summary calculation result, and the verification data block into the business streaming data packet corresponding to the business data.

[0124] The sending module 304 is adapted to send the inserted service streaming data packets to the data receiver.

[0125] Optionally, the service streaming data packet includes: a first service streaming data packet and a second service streaming data packet;

[0126] The insertion module is further adapted to insert the verification file name, the verification file read offset, a portion of the first digest calculation result, and the verification data block into the first business streaming data packet corresponding to the business data;

[0127] Insert the verification file name, verification file read offset, and verification data block into the second business streaming data packet corresponding to the business data.

[0128] Optionally, the insertion module is further adapted to: take the remainder of the length of the first digest calculation result using the offset read from the verification file, and determine the value of a specified bit of the first digest calculation result based on the first remainder value;

[0129] Insert the verification file name, verification file read offset, the value of a specified bit of the first digest calculation result, and the verification data block into the business streaming data packet corresponding to the business data.

[0130] Optionally, the apparatus further includes: a calculation module, adapted to calculate a second remainder value using formula (1) if the length of the verification file name is greater than the length of the first digest calculation result:

[0131] Second remainder value = (check file read offset / check data block length) % Check file name length formula (1);

[0132] The insertion module is further adapted to: if the second remainder value is equal to 0, then determine multiple consecutive first service streaming data packets corresponding to the service data; for each first service streaming data packet, take the remainder of the length of the first digest calculation result using the check file read offset; determine the value of a specified bit of the first digest calculation result based on the third remainder value; and insert the check file name, check file read offset, the value of the specified bit of the first digest calculation result, and the check data block into the first service streaming data packet respectively.

[0133] If the second remainder value is not equal to 0, then the verification file name, the verification file read offset, and the verification data block are inserted into the second business streaming data packet corresponding to the business data.

[0134] If the length of the verification file name is less than or equal to the length of the first digest calculation result, then the verification file name, the verification file read offset, the entire first digest calculation result, and the verification data block are inserted into the respective business streaming data packets corresponding to the business data.

[0135] Optionally, the apparatus further includes: a processing module, adapted to obtain at least one first file to be verified according to a pre-configured verification file directory;

[0136] Perform digest calculation on at least one first document to be verified to obtain the first digest calculation result of the first document to be verified.

[0137] Store the verification file name, first digest calculation result, and verification file read offset corresponding to at least one first file to be verified into a circular linked list;

[0138] The acquisition module is further adapted to: retrieve the verification file information of the current first file to be verified from the circular linked list.

[0139] Optionally, the reading module is further adapted to: determine whether the file reading offset is the file end offset;

[0140] If so, initialize the verification file read offset and retrieve the verification file information of the next first file to be verified from the circular linked list;

[0141] If not, then read a preset size of verification data block from the current first file to be verified based on the verification file read offset.

[0142] Optionally, the apparatus further includes an update module adapted to update the check file read offset according to the size of the check data block.

[0143] The solution provided by this invention has a natural integrity verification method for the verification file. A deterministic result is obtained by performing a digest calculation on the verification file. Modifying any data in the verification file will cause the digest calculation result to change. Therefore, based on the mechanism of the data sender sending verification file information (verification file name, verification file read offset, and all or part of the first digest calculation result), problems such as sending omissions or failures can be eliminated, thereby improving the accuracy of data stream integrity verification.

[0144] Figure 4 A schematic diagram of the structure of the streaming data integrity verification device provided in an embodiment of the present invention is shown. Figure 4 As shown, the device includes: a receiving module 401, an extraction module 402, a writing module 403, a calculation module 404, a matching module 405, and a determining module 406.

[0145] Receiver module 401 is adapted to receive service streaming data packets sent by the data sender;

[0146] Extraction module 402 is adapted to extract the verification file name, verification file read offset, verification data block, and first digest calculation result from at least one business streaming data packet;

[0147] The writing module 403 is adapted to write the verification data block to the second file to be verified corresponding to the verification file name according to the read offset of the verification file;

[0148] The calculation module 404 is adapted to perform a digest calculation on the second file to be verified if no business streaming data packet is received within a preset time period, and obtain the second digest calculation result.

[0149] The matching module 405 is adapted to match the second summary calculation result with the first summary calculation result;

[0150] Module 406 is adapted to determine the data integrity verification result based on the matching result.

[0151] Optionally, the calculation module is further adapted to: count the duration during which no business streaming data packets sent by the data sender are received;

[0152] If the statistical duration is greater than or equal to the preset duration, a digest calculation is performed on the second file to be verified to obtain the second digest calculation result.

[0153] Optionally, the extraction module is further adapted to: if the length of the verified file name is greater than the length of the first digest calculation result, then calculate the fourth remainder value using formula (2):

[0154] Fourth remainder value = (check file read offset / check data block length)% Check file name length formula (2);

[0155] If the fourth remainder value is equal to 0, then extract the check file name, check file read offset, the value of a specified bit of the first digest calculation result, and the check data block from the business streaming data packet.

[0156] If the fourth remainder value is not equal to 0, then extract the verification file name, verification file read offset, and verification data block from the business streaming data packet;

[0157] If the length of the verification file name is less than or equal to the length of the first digest calculation result, then the verification file name, verification file read offset, verification data block, and first digest calculation result are extracted from at least one business streaming data packet.

[0158] Optionally, the apparatus further includes: a remainder module, adapted to perform a remainder on the length of the first digest calculation result using a check file read offset when extracting the value of a specified bit of the first digest calculation result from the service streaming data packet, and to store the value of the specified bit of the first digest calculation result into a specified location according to a fifth remainder value, so as to assemble the first digest calculation result.

[0159] Optionally, the writing module is further adapted to: if the first digest calculation result is not assembled, write the verification data block to the second file to be verified corresponding to the verification file name according to the verification file read offset;

[0160] If the first digest calculation result is obtained by assembly, the second file to be verified is renamed according to the verification file name and the first digest calculation result to generate a new verification file name. The verification data block is written to the second file to be verified corresponding to the new verification file name according to the verification file read offset.

[0161] The solution provided by this invention, upon receiving a service streaming data packet from a data sender, extracts the first digest calculation result and writes the data in the verification data block into a second file to be verified. During integrity verification, a second digest calculation result is generated for the second file to be verified, and the second digest calculation result is matched with the first digest calculation result. The data integrity verification result is determined based on the matching result, thereby accurately determining whether the received service data has been lost. Furthermore, by transmitting verification file information, the trust between end-to-end nodes is increased. The verification file can be transmitted to each end-to-end node, and each node can perform integrity checks based on the verification file information.

[0162] This invention provides a non-volatile computer storage medium storing at least one executable instruction that can execute the method in any of the above-described method embodiments.

[0163] Figure 5 The diagram shows a structural schematic of a computing device provided in an embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computing device.

[0164] like Figure 5 As shown, the computing device may include a processor, a communications interface, memory, and a communications bus.

[0165] The processor, communication interface, and memory communicate with each other via a communication bus. The communication interface is used to communicate with other network elements, such as clients or other servers. The processor executes programs, specifically the relevant steps in any of the above-described method embodiments for computing devices.

[0166] Specifically, the program may include program code, which includes computer operation instructions.

[0167] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computing device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0168] Memory is used to store programs. Memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive.

[0169] Specifically, the program can be used to cause the processor to execute the methods in any of the above method embodiments. The specific implementation of each step in the program can be found in the corresponding descriptions of the steps and units in any of the above embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0170] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the embodiments of the present invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the embodiments of the present invention.

[0171] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0172] Similarly, it should be understood that, in order to streamline the embodiments of the invention and aid in understanding one or more of the various inventive aspects, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed embodiments of the invention require more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0173] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0174] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0175] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The embodiments of the present invention can also be implemented as device or apparatus programs (e.g., computer programs and computer program products) for performing part or all of the methods described herein. Such programs implementing the embodiments of the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0176] It should be noted that the above embodiments are illustrative of the present invention and not restrictive of the invention, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. Embodiments of the present invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A method for transmitting streaming data, comprising: Obtain the verification file information corresponding to the current first file to be verified, wherein the verification file information includes: verification file name, verification file read offset, and first digest calculation result; A preset size of verification data block is read from the current first file to be verified according to the verification file read offset; Insert the verification file name, verification file read offset, all or part of the first digest calculation result, and the verification data block into the business streaming data packet corresponding to the business data, and send the inserted business streaming data packet to the data receiver; The service streaming data packet includes: a first service streaming data packet and a second service streaming data packet; Inserting the verification file name, verification file read offset, part of the first digest calculation result, and the verification data block into the business streaming data packet corresponding to the business data further includes: Insert the verification file name, the verification file read offset, a portion of the first digest calculation result, and the verification data block into the first service streaming data packet corresponding to the service data; The method further includes: inserting the verification file name, the verification file read offset, and the verification data block into the second service streaming data packet corresponding to the service data; After reading a preset-size verification data block from the current first file to be verified according to the verification file read offset, the method further includes: If the length of the verification file name is greater than the length of the first digest calculation result, then the second remainder value is calculated using formula (1): The step of inserting the verification file name, the verification file read offset, a portion of the first digest calculation result, and the verification data block into the first service streaming data packet corresponding to the service data further includes: If the second remainder value is equal to 0, then multiple consecutive first service streaming data packets corresponding to the service data are determined. For each first service streaming data packet, the length of the first digest calculation result is moduloed by the verification file read offset. The value of a specified bit of the first digest calculation result is determined according to the third remainder value. The verification file name, verification file read offset, the value of the specified bit of the first digest calculation result, and the verification data block are respectively inserted into the first service streaming data packet. The step of inserting the verification file name, the verification file read offset, and the verification data block into the second service streaming data packet corresponding to the service data further includes: If the second remainder value is not equal to 0, then the verification file name, the verification file read offset, and the verification data block are inserted into the second service streaming data packet corresponding to the service data. If the length of the verification file name is less than or equal to the length of the first digest calculation result, then the verification file name, the verification file read offset, the entire first digest calculation result, and the verification data block are inserted into each business streaming data packet corresponding to the business data.

2. The method according to claim 1, wherein, The step of inserting the verification file name, the verification file read offset, a portion of the first digest calculation result, and the verification data block into the service streaming data packet corresponding to the service data further includes: The offset read from the verification file is used to take the remainder of the length of the first digest calculation result, and the value of a specified bit in the first digest calculation result is determined based on the first remainder value. The verification file name, the verification file read offset, the value of a specified bit of the first digest calculation result, and the verification data block are inserted into the business streaming data packet corresponding to the business data.

3. The method according to claim 1, wherein, Before obtaining the verification file information corresponding to the current first file to be verified, the method further includes: Based on the pre-configured verification file directory, obtain at least one first file to be verified; Perform digest calculation on the at least one first document to be verified to obtain a first digest calculation result of the first document to be verified. Store the verification file name, first digest calculation result, and verification file read offset corresponding to at least one first file to be verified into a circular linked list; The step of obtaining the verification file information corresponding to the current first file to be verified further includes: Obtain the verification file information of the first file to be verified from the circular linked list.

4. The method according to claim 3, wherein, The step of reading a preset size of verification data block from the current first file to be verified according to the verification file read offset further includes: Determine whether the file read offset is the offset from the end of the file; If so, the offset for reading the verification file is initialized, and the verification file information of the next first file to be verified is obtained from the circular linked list; If not, then a preset size of verification data block is read from the current first file to be verified according to the verification file read offset.

5. The method according to claim 1, wherein, After the insertion is completed, the method further includes updating the check file read offset according to the size of the check data block.

6. A streaming data integrity verification method, comprising: Receive service streaming data packets sent by a data sender, wherein the service streaming data packets are data packets sent based on any of the streaming data sending methods in claims 1-5; Extract the verification file name, verification file read offset, verification data block, and first digest calculation result from at least one business streaming data packet; The verification data block is written to the second file to be verified corresponding to the verification file name according to the verification file read offset. If no business streaming data packet is received within the preset time period, a digest calculation is performed on the second file to be verified to obtain the second digest calculation result; The result of the second summary calculation is matched with the result of the first summary calculation; Based on the matching results, determine the data integrity verification result.

7. The method according to claim 6, wherein, If no service streaming data packet is received within a preset time period, then performing a digest calculation on the second file to be verified to obtain the second digest calculation result further includes: Statistical duration of business streaming data packets not received from the data sender; If the statistical duration is greater than or equal to the preset duration, a digest calculation is performed on the second file to be verified to obtain the second digest calculation result.

8. The method according to claim 6, wherein, Extracting the checksum file name, checksum file read offset, checksum data block, and first digest calculation result from at least one business streaming data packet further includes: If the length of the verification file name is greater than the length of the first digest calculation result, then the fourth remainder value is calculated using formula (2): If the fourth remainder value is equal to 0, then extract the verification file name, verification file read offset, the value of a specified bit of the first digest calculation result, and the verification data block from the business streaming data packet. If the fourth remainder value is not equal to 0, then extract the verification file name, verification file read offset and verification data block from the business streaming data packet; If the length of the verification file name is less than or equal to the length of the first digest calculation result, then the verification file name, verification file read offset, verification data block, and first digest calculation result are extracted from at least one business streaming data packet.

9. The method according to claim 8, wherein, For the case of extracting the value of a specified bit of the first digest calculation result from the business streaming data packet, the method further includes: taking the remainder of the length of the first digest calculation result using the offset read from the verification file, and storing the value of the specified bit of the first digest calculation result into a specified location according to the fifth remainder value, so as to assemble the first digest calculation result.

10. The method according to claim 9, wherein, The step of writing the verification data block to the second file to be verified corresponding to the verification file name based on the verification file read offset further includes: If the first digest calculation result is not obtained by assembly, the verification data block is written to the second file to be verified corresponding to the verification file name according to the verification file read offset; If the first digest calculation result is obtained by assembly, the second file to be verified is renamed according to the verification file name and the first digest calculation result to generate a new verification file name. The verification data block is written to the second file to be verified corresponding to the new verification file name according to the verification file read offset.

11. A streaming data transmission device, comprising: The acquisition module is adapted to acquire the verification file information corresponding to the current first file to be verified, wherein the verification file information includes: verification file name, verification file read offset, and first digest calculation result; The reading module is adapted to read a preset size of verification data block from the current first file to be verified according to the verification file reading offset; The insertion module is adapted to insert the verification file name, the verification file read offset, all or part of the first digest calculation result, and the verification data block into the business streaming data packet corresponding to the business data; The sending module is suitable for sending the inserted business streaming data packets to the data receiver; The service streaming data packet includes: a first service streaming data packet and a second service streaming data packet; The insertion module is further adapted to insert the verification file name, the verification file read offset, a portion of the first digest calculation result, and the verification data block into the first business streaming data packet corresponding to the business data; Insert the verification file name, verification file read offset, and verification data block into the second business streaming data packet corresponding to the business data; The calculation module is suitable for calculating the second remainder value using formula (1) if the length of the verification file name is greater than the length of the first digest calculation result: The insertion module is further adapted to: if the second remainder value is equal to 0, then determine multiple consecutive first service streaming data packets corresponding to the service data; for each first service streaming data packet, take the remainder of the length of the first digest calculation result using the check file read offset; determine the value of a specified bit of the first digest calculation result based on the third remainder value; and insert the check file name, check file read offset, the value of the specified bit of the first digest calculation result, and the check data block into the first service streaming data packet respectively. If the second remainder value is not equal to 0, then the verification file name, the verification file read offset, and the verification data block are inserted into the second business streaming data packet corresponding to the business data. If the length of the verification file name is less than or equal to the length of the first digest calculation result, then the verification file name, the verification file read offset, the entire first digest calculation result, and the verification data block are inserted into the respective business streaming data packets corresponding to the business data.

12. A streaming data integrity verification device, comprising: A receiving module is adapted to receive service streaming data packets sent by a data sender, wherein the service streaming data packets are data packets sent based on the streaming data sending device according to claim 11; The extraction module is suitable for extracting the verification file name, verification file read offset, verification data block, and first digest calculation result from at least one business streaming data packet. The writing module is adapted to write the verification data block to the second file to be verified corresponding to the verification file name according to the verification file read offset; The calculation module is adapted to perform a digest calculation on the second file to be verified if no business streaming data packet is received within a preset time period, and obtain a second digest calculation result; The matching module is adapted to match the second summary calculation result with the first summary calculation result; The determination module is suitable for determining the streaming data integrity verification result based on the matching result.

13. A computing device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the streaming data transmission method as described in any one of claims 1-5 or to perform the operation corresponding to the streaming data integrity verification method as described in any one of claims 6-10.

14. A computer storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to the streaming data transmission method as described in any one of claims 1-5 or the streaming data integrity verification method as described in any one of claims 6-10.

Images