File verification transmission method, device, equipment and storage based on Kafka

Abstract

The present disclosure provides a file verification transmission method, device, equipment and storage based on Kafka, which relates to the field of big data and data security technology. The method includes: obtaining multiple file fragments by fragmenting the original file; sending the identification information of the multiple file fragments to the middleware, so that the middleware creates a one-to-one corresponding file fragment topic according to the identification information of the multiple file fragments; cutting each file fragment into multiple message slices and sending them to the middleware, so that the middleware stores the message slices according to the file fragment topic corresponding to the message slice; pulling each message slice in the middleware and comparing and verifying it with the message slice in the corresponding file fragment, and when the comparison and verification are consistent, determining that the message slice in the middleware is the first message slice. The present disclosure compares and verifies the message slice in the middleware with the message slice in the corresponding file fragment, thereby ensuring the reliability of the message from the sender to the middleware.

Description

Background Art

[0002] Kafka is a distributed publish-subscribe messaging system, mainly composed of producers, brokers, and consumers. The producer is responsible for generating messages and sending them to the broker, which temporarily stores the messages and then forwards them to the consumer, who receives and processes the messages. Kafka is efficient when transmitting small messages, but in some application scenarios where large files need to be transmitted, such as transmitting videos and large archive files, a lot of network resources are wasted on sending and retransmitting, resulting in low message transmission rate and efficiency.

[0003] In the prior art, the original file is divided into small messages for transmission, but the transmitted small messages cannot be verified, resulting in unreliable data transmission.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides a file verification transmission method, device, equipment and storage based on Kafka, which at least to a certain extent overcomes the problem that small messages based on Kafka file segmentation in related technologies cannot be verified.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a file verification transmission method based on Kafka is provided, which is applied to a sending end, and includes: slicing an original file to obtain multiple file slicings; sending identification information of the multiple file slicings to a middleware, so that the middleware creates one-to-one corresponding file slicing topics according to the identification information of the multiple file slicings; cutting each file slicing into multiple message slices and sending them to the middleware, so that the middleware stores the message slices according to the file slicing topics corresponding to the message slices, wherein the message slices include the identification information of the corresponding file slicing; pulling each message slice in the middleware and comparing it with the message slice in the corresponding file slicing for verification, and when the comparison and verification are consistent, determining that the message slice in the middleware is a first message slice, wherein the first message slice represents the message slice transmitted from the sending end to the middleware and passed the verification.

[0008] In one embodiment of the present disclosure, each message slice in the middleware is pulled and compared with the message slice in the corresponding file slice for verification, including: when the comparison and verification are inconsistent, resending the inconsistent message slice to the middleware.

[0009] In one embodiment of the present disclosure, fragmenting an original file includes: acquiring the original file and determining file meta information of the original file; and fragmenting the original file according to the file meta information to obtain a plurality of file fragments.

[0010] In one embodiment of the present disclosure, after obtaining the original file and determining the file meta information of the original file, the method further includes: sending the file meta information to a middleware, so that a receiving end obtains the file meta information through the middleware for combining file fragments.

[0011] According to another aspect of the present disclosure, a Kafka-based file verification transmission method is provided, which is applied to a receiving end, and includes: pulling multiple first message slices in the middleware and combining them to obtain multiple file slices, wherein the first message slices represent message slices that are transmitted from the sending end to the middleware and have passed verification; pulling file meta-information, combining each file slice according to the file meta-information, and determining a combined file; verifying the MD5 value of the combined file with the MD5 value of the original file, and when the verification is consistent, determining that the combined file is the original file of the sending end, and completing file verification reception.

[0012] According to another aspect of the present disclosure, a file verification transmission system based on Kafka is provided, including: a sending end, a middleware, and a receiving end; the sending end is used to slice the original file to obtain multiple file slices; the identification information of the multiple file slices is sent to the middleware; each file slice is cut into multiple message slices and sent to the middleware; each message slice in the middleware is pulled and compared with the message slice in the corresponding file slice for verification, and when the comparison and verification are consistent, the message slice in the middleware is determined to be the first message slice; the middleware is used to receive the identification information of the multiple file slices sent by the sending end, and create a one-to-one corresponding file according to the identification information of the multiple file slices. The receiving end is used to pull multiple first message slices in the middleware and combine them to obtain multiple file slices; pull file meta information, combine each file slice according to the file meta information, and determine the combined file; verify the MD5 value of the combined file with the MD5 value of the original file, and when the verification is consistent, determine that the combined file is the original file of the sending end, and complete the file reception verification; wherein, the message slice includes the identification information of the corresponding file slice, and the first message slice represents the message slice transmitted by the sending end to the middleware and passed the verification.

[0013] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the above-mentioned Kafka-based file verification and transmission method by executing the executable instructions.

[0014] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned Kafka-based file verification and transmission method is implemented.

[0015] According to another aspect of the present disclosure, a computer program product is provided, including computer instructions, wherein the computer instructions are stored in a computer-readable storage medium, and when the computer instructions are executed by a processor, the operating instructions of any of the above-mentioned Kafka-based file verification and transmission methods are implemented.

[0016] The Kafka-based file verification transmission method, device, equipment and storage provided by the embodiments of the present disclosure are applied to the sending end, and multiple file fragments are obtained by fragmenting the original file; the identification information of the multiple file fragments is sent to the middleware, so that the middleware creates a one-to-one corresponding file fragment topic according to the identification information of the multiple file fragments; each file fragment is divided into multiple message slices and sent to the middleware, so that the middleware stores the message slices according to the file fragment topic corresponding to the message slice, wherein the message slice includes the identification information of the corresponding file fragment; each message slice in the middleware is pulled and compared with the message slice in the corresponding file fragment for verification, and when the comparison and verification are consistent, the message slice in the middleware is determined to be the first message slice, wherein the first message slice represents the message slice transmitted by the sending end to the middleware through verification. The present disclosure ensures the reliability of the message from the sending end to the kafka message middleware by comparing and verifying the message slice in the middleware with the message slice in the corresponding file fragment.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0019] Figure 1A schematic diagram showing the system structure of a Kafka-based file verification transmission method in an embodiment of the present disclosure;

[0020] Figure 2 A schematic diagram showing a specific example of a system architecture of a file verification transmission method based on Kafka in an embodiment of the present disclosure;

[0021] Figure 3 A flow chart of a Kafka-based file verification transmission method applied to a sending end in an embodiment of the present disclosure is shown;

[0022] Figure 4 A flowchart of a Kafka-based file verification transmission method applied to a sending end in an embodiment of the present disclosure is shown;

[0023] Figure 5 A flowchart of a file verification transmission method based on Kafka applied to a sending end in an embodiment of the present disclosure is shown;

[0024] Figure 6 Another flowchart of a Kafka-based file verification transmission method applied to a sending end in an embodiment of the present disclosure is shown;

[0025] Figure 7 A flow chart of a Kafka-based file verification transmission method applied to a receiving end in an embodiment of the present disclosure is shown;

[0026] Figure 8 A flowchart showing a specific example of a Kafka-based file verification transmission method applied to a sending end in an embodiment of the present disclosure;

[0027] Fig. 9 A flowchart showing another specific example of a Kafka-based file verification transmission method applied to a sending end in an embodiment of the present disclosure;

[0028] Fig.10 A flowchart showing a specific example of a Kafka-based file verification transmission method applied to a receiving end in an embodiment of the present disclosure;

[0029] Fig.11 A flowchart showing another specific example of a Kafka-based file verification transmission method applied to a receiving end in an embodiment of the present disclosure;

[0030] Fig.12 A schematic diagram of message slice splicing in an embodiment of the present disclosure is shown;

[0031] Fig.13 A schematic diagram of file segment splicing in an embodiment of the present disclosure is shown;

[0032] Fig.14A schematic diagram of a Kafka-based file verification transmission device applied to a sending end in an embodiment of the present disclosure is shown;

[0033] Fig.15 A schematic diagram of a Kafka-based file verification transmission device applied to a receiving end in an embodiment of the present disclosure is shown;

[0034] Fig.16 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0035] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0036] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0037] Figure 1 A schematic diagram of an exemplary system architecture of a Kafka-based file verification transmission method or a Kafka-based file verification transmission device that can be applied to an embodiment of the present disclosure is shown.

[0038] like Figure 1 As shown, the system architecture 100 may include a sending end 101 , a middleware 102 , a receiving end 103 and a network 104 .

[0039] The network 104 is used as a medium to provide a communication link between the sending end 101, the middleware 102 and the receiving end 103, and can be a wired network or a wireless network.

[0040] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged through the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0041] The sending end 101 is used to slice the original file to obtain multiple file slices; send the identification information of the multiple file slices to the middleware; cut each file slice into multiple message slices and send them to the middleware; pull each message slice in the middleware and compare and verify it with the message slice in the corresponding file slice, and when the comparison and verification are consistent, determine that the message slice in the middleware is the first message slice; the middleware 102 is used to receive the identification information of the multiple file slices sent by the sending end, create a one-to-one corresponding file slice topic according to the identification information of the multiple file slices; receive the multiple message slices sent by the sending end, and create a one-to-one corresponding file slice topic according to the file slice corresponding to the message slice The file segmentation topic stores message slices; the receiving end 103 is used to pull multiple first message slices in the middleware and combine them to obtain multiple file slices; pull file meta information, combine each file slice according to the file meta information, and determine the combined file; verify the MD5 value of the combined file with the MD5 value of the original file, and when the verification is consistent, determine that the combined file is the original file of the sender, and complete the file reception verification; wherein, the message slice includes the identification information of the corresponding file slice, and the first message slice represents the message slice transmitted from the sender to the middleware and verified by each message slice in the sender and each corresponding message slice in the middleware.

[0042] Those skilled in the art will know that Figure 1 The number of the sending end, the middleware, the receiving end and the network is only for illustration, and any number of the sending end, the middleware, the receiving end and the network may be provided according to actual needs. The embodiments of the present disclosure are not limited to this.

[0043] Figure 2 A schematic diagram of a specific example of an exemplary system architecture that can be applied to the Kafka-based file verification transmission method or the Kafka-based file verification transmission device of the embodiments of the present disclosure is shown.

[0044] like Figure 2 As shown, the system architecture 200 may include a sender 201 , a Kafka middleware 202 , and a receiver 203 .

[0045] The sender 201 includes a file meta information generation module 2010 and a thread pool 210, and the thread pool 210 includes a first producer 221, a second producer 222, and a third producer 223. The Kafka middleware 202 includes a file meta information Topic module 2020, a file shard Topic1 module 2021, a file shard Topic2 module 2022, and a file shard TopicN module 2023. The receiver 203 includes a file meta information pulling Topic module 2030 and a file shard verification and combination module 230.

[0046] When the system is running, the sender 201 divides the original file into multiple file slices, for example, the original file is divided into file slice 211, file slice 212, and file slice 213. The file slices are divided into multiple message slices. The message slices are sent and pulled and verified by the producer thread and stored in the file slice content Topic. The file slice metadata message is sent to the metadata Topic. The receiver 203 pulls the specified Topic content based on the metadata Topic content. The message slices are compacted and combined into file slices. The file slices are verified and then combined into the original file.

[0047] Among them, the sender reads the original file and performs file sharding according to the set file shard size. The sender creates a thread pool, and multiple threads send file sharding information to the Kafka message middleware in parallel. The producer reads the file sharding and sends the file sharding content to the file sharding content Topic of the message Kafka message middleware in a message of 10Kb size. When the file sharding content is sent, the producer will continue to pull messages from the file sharding content Topic to verify whether the file sharding content on the Kafka middleware is consistent with the content of the sender. If they are consistent, the Topic name, Topic start bit, Topic end bit, file sharding MD5 value and other information are saved, and the thread ends. If they are inconsistent, this proves that an error occurred during the transmission process, and the producer resends the file sharding content. When all threads end, the sender adds the original file name, number of file shards, file MD5 value and each file shard to the file meta information Topic.

[0048] The Kafka message middleware in the present invention has two types of topics:

[0049] One of them is to save the original file metadata information sent by the sender. The specific data structure is shown in Table 1 below: The message of this Topic contains the original file name, the original file MD5 value, the number of original file fragments and the file fragment Topic name, the Topic start position, the Topic end position, and the set of file fragment MD5 values.

[0050] Table 1

[0051]

[0052]

[0053] The MD5 value of the original file is a value generated by the MD5 algorithm when reading the original file. It is used to verify whether the original file is consistent with the received original file. The number of original file shards is the rounded-up ratio of the original file size to the file shard size set by the user. The default file shard topic names are Topic1, Topic2-TopicN. The Topic start bit is the maximum value of the Topic message offset that the producer starts to send to the Kafka message middleware. The Topic end bit is the maximum value of the Topic message offset after the producer pulls the information from the Kafka message middleware and verifies it. The file shard MD5 value is a value generated by the MD5 algorithm when reading the file shard. It is used to verify whether the file shard sent by the sender to the Kafka middleware is correct.

[0054] The other is to save the content Topic of the file fragments split by the sender, and save the file fragment information sent by the producer in the producer thread pool. The specific data structure is shown in Table 2 below: The Topic message structure consists of a message header and a message slice. The message header includes a message identification code, the total number of file fragments, and the number of file fragments.

[0055] Table 2

[0056]

[0057] The message identification code identifies which file segment the message belongs to. The total number of file segments is the number of file segments divided by the value of message segments. The number of file segments is the segment value of each message segment sent. The message slice is the segment content of the file segment.

[0058] When the system is running, the receiver pulls the file metadata Topic and file slice Topic information in sequence. The receiver first pulls the metadata Topic content, and then pulls the content from the start bit to the end bit of each Topic in sequence according to the content. In the process of splicing file fragments with messages. When encountering messages with the same message header, the message with the older timestamp is overwritten with the message with the newer timestamp to cover the incorrect message. Then the extracted Topic content is verified by the MD5 value. If the verification passes, continue to pull the content of the next Topic. If the verification fails, pull the content of the Topic again. After pulling all the file slice content Topics, combine the file fragments. Then verify whether the MD5 value of the combined file is consistent with the MD5 value of the original file. If they are consistent, the file received by the receiver is consistent with the original file of the sender.

[0059] The present invention ensures the data reliability between the sender and the Kafka message middleware through the producer pull verification method of the sender. The data reliability from the Kafka message middleware to the receiver is ensured by the receiver using the file segment MD5 value to verify the pulled file segment. This ensures that the original file from the sender is completely consistent with the combination of the receiver, and achieves the end-to-end reliability of message transmission; the sender sends the valid start bit and end bit of the Topic message. The receiver can accurately pull the valid file segment content Topic information. The efficient pulling of the receiver's information is achieved; through the sending, pulling, verification and retransmission mechanism of the sender's message slice, when the verification error occurs and the message is retransmitted, the network bandwidth is saved and the efficiency of file segment transmission is improved.

[0060] The present exemplary implementation is described in detail below with reference to the accompanying drawings and embodiments.

[0061] The present disclosure provides a Kafka-based file verification and transmission method, which can be executed by any electronic device with computing and processing capabilities.

[0062] Figure 3 A flow chart of a Kafka-based file verification transmission method applied to a sending end in an embodiment of the present disclosure is shown as follows: Figure 3 As shown, the Kafka-based file verification transmission method provided in the embodiment of the present disclosure includes the following steps:

[0063] S302, fragment the original file to obtain multiple file fragments.

[0064] It should be noted that the original file may be a large file, such as a video file, a large archive file, etc. The file segmentation may be to segment the original file into a plurality of small segments.

[0065] S304, sending identification information of the multiple file segments to the middleware, so that the middleware creates a one-to-one corresponding file segment topic according to the identification information of the multiple file segments.

[0066] It should be noted that the above identification information may be a file name.

[0067] For example, in one example, multiple file segments and their corresponding file names are sent to the middleware, so that the middleware creates one-to-one corresponding file segment topics according to the file names of the multiple file segments. For example, file segment a, file segment b, and file segment c correspond to file segment topic a, file segment topic b, and file segment topic c in the middleware, respectively.

[0068] S306, split each file slice into multiple message slices and send them to the middleware, so that the middleware stores the message slices according to the file slice topics corresponding to the message slices, wherein the message slices include identification information of the corresponding file slices.

[0069] It should be noted that the message slices can be obtained by dividing the file into segments, and are smaller data than the file segments. The middleware can be a container for storing messages during the transmission of the messages.

[0070] For example, in one instance, file shard a is divided into message slice a1 and message slice a2. Since file shard a corresponds to file shard topic a, message slice a1 and message slice a2 will both be stored in file shard topic a.

[0071] S308, pull each message slice in the middleware and compare and verify it with the message slice in the corresponding file slice. When the comparison and verification are consistent, determine that the message slice in the middleware is the first message slice, wherein the first message slice represents the message slice transmitted by the sender to the middleware and passed the verification.

[0072] For example, in one instance, the sending end pulls the message slice a1 and message slice a2 stored in the middleware file sharding topic a, and compares them with the message slice a1 and message slice a2 cut from the file shard a in the sending end. If the two groups of message slices are found to be the same, it is determined that the message slices a1 and message slice a2 in the middleware are both the first message slices, and the sending end completes the sending and verification of the original file.

[0073] The present disclosure ensures the reliability of messages from the sender to the Kafka message middleware by comparing and verifying the message slices in the middleware with the message slices in the corresponding file slices.

[0074] In one embodiment of the present disclosure, Figure 4 As shown, the Kafka-based file verification transmission method provided in the embodiment of the present disclosure can resend message slices through the following steps, and can accurately detect the message slices that are sent incorrectly in the message slices:

[0075] S402, when the verification content is inconsistent, resend the verification inconsistent message slice to the middleware.

[0076] For example, in one instance, the sending end pulls the message slice a1 and the message slice a2 stored in the middleware file sharding topic a, and compares them with the message slice a1 and the message slice a2 segmented from the file shard a in the sending end. After comparison, the message slice a1 stored in the file sharding topic a and the message slice a1 segmented from the file shard a in the sending end are not exactly the same. Then, the sending end resends the message slice a1 segmented from the file shard a to the middleware file sharding topic a, overwriting the message slice a1 stored in the middleware file sharding topic a.

[0077] The present invention compares and verifies the message slices in the middleware with the message slices in the corresponding file slices. When an error occurs, the message slice with the transmission error is resent without the need to re-upload all the message slices. This not only ensures the message reliability from the sender to the Kafka message middleware, but also saves network bandwidth and improves the efficiency of file slice transmission.

[0078] In one embodiment of the present disclosure, Figure 5 As shown, the Kafka-based file verification transmission method provided in the embodiment of the present disclosure can perform sharding processing on the original file through the following steps, and can accurately obtain multiple file shards:

[0079] S502, obtaining an original file and determining file meta information of the original file;

[0080] S504: Segment the original file according to the file meta-information to obtain multiple file segments.

[0081] It should be noted that the file meta information is equivalent to the above-mentioned original file metadata information, including the original file name, original file MD5 value, original file fragment number and file fragment Topic name, Topic start bit, Topic end bit, and file fragment MD5 value set.

[0082] In one embodiment of the present disclosure, Figure 6 As shown, the Kafka-based file verification transmission method provided in the embodiment of the present disclosure can process the file meta information through the following steps, so that the receiving end can combine the file fragments according to the file meta information:

[0083] S602, sending the file meta information to the middleware, so that the receiving end obtains the file meta information through the middleware for combining the file segments.

[0084] Figure 7 A flow chart of a Kafka-based file verification transmission method applied to a receiving end in an embodiment of the present disclosure is shown. Figure 7 As shown, the Kafka-based file verification transmission method provided in the embodiment of the present disclosure includes the following steps:

[0085] S702, pull multiple first message slices in the middleware and combine them to obtain multiple file slices, wherein the first message slices represent message slices that are transmitted from the sender to the middleware and passed verification.

[0086] For example, multiple message slices that have completed verification in the pull middleware (equivalent to the above-mentioned first message slice), wherein the first message slice is processed by the sending end by slicing the original file to obtain multiple file slices, and the identification information of the multiple file slices is sent to the middleware, so that the middleware creates a one-to-one corresponding file slice topic according to the identification information of the multiple file slices, and divides each file slice into multiple message slices and sends them to the middleware, so that the middleware stores the message slices according to the file slice topic corresponding to the message slice, wherein the message slice includes the identification information of the corresponding file slice, and each message slice in the pull middleware is compared and verified with the message slice in the corresponding file slice. When the comparison and verification are consistent, it is determined that the message slice in the middleware is the first message slice, and the first message slice represents the message slice transmitted by the sending end to the middleware and passed the verification.

[0087] S704, pull file meta information, combine each file segment according to the file meta information, and determine a combined file.

[0088] It should be noted that the file meta information is equivalent to the above-mentioned original file metadata information, including the original file name, original file MD5 value, original file fragment number and file fragment Topic name, Topic start bit, Topic end bit, and file fragment MD5 value set.

[0089] S706, verifying the MD5 value of the combined file and the MD5 value of the original file. When the verification is consistent, the combined file is determined to be the original file of the sender, and the file reception verification is completed.

[0090] The receiving end of the present invention verifies through the MD5 value of the file fragment, realizes the verification from the Kafka message middleware to the receiving end, and realizes the consistent reliability from the sender's original file to the receiver's combined file by ensuring the consistency from the middleware to the receiving end.

[0091] Figure 8 A flowchart of a specific example of a Kafka-based file verification transmission method applied to a sending end in an embodiment of the present disclosure is shown as follows: Figure 8 As shown, the Kafka-based file verification transmission method for sending an original file provided by the sender in the embodiment of the present disclosure includes the following steps:

[0092] S801, read the file;

[0093] S802, generate file meta information, and calculate the number of threads N according to the file size;

[0094] S803, generating a thread pool with a thread capacity of N;

[0095] S804, start N threads and send file segments;

[0096] S805, query whether the task in the thread pool has been executed. If not, wait for 100ms and continue to query; if yes, the execution is completed, then go to S806;

[0097] S806, sending information to the file metadata Topic, and the program ends.

[0098] Fig. 9 A flowchart showing another specific example of a Kafka-based file verification transmission method applied to a sending end in an embodiment of the present disclosure is shown as follows: Fig. 9 As shown, the Kafka-based file verification transmission method for sending file fragments provided by the sender in the embodiment of the present disclosure includes the following steps:

[0099] S901, thread N reads file slice N;

[0100] S902, thread N creates a message fragmenter, producer, message fragmentation, and saves the TopicN message offset as the TopicN starting amount;

[0101] S903, verify whether the message fragment has been sent. If it has been sent, go to S905; if it has not been sent, go to S904;

[0102] S904, continue to send the message fragment to the topic TopicN, go to S903, and continue to verify whether the message fragment has been sent;

[0103] S905, pull TopicN topic information;

[0104] S906, determine whether the pulled message is empty, if yes, it is empty, go to S912; if no, it is not empty, go to S907;

[0105] S907, verify whether the message is correct. If yes, go to S905 and continue to pull the Topic N message; if no, go to S908;

[0106] S908, obtaining the value of the message fragment;

[0107] S909, determining the displacement start and end values ​​of the message fragment in the file according to the fragment value;

[0108] S910, repeatedly fragment the message to Topic N, go to S905, and continue to pull the message in Topic N;

[0109] S911, verify whether the message is complete, if not, the message is incomplete, go to S908; if yes, the message is complete, go to S912;

[0110] S912, read the TopicN message offset at this time as the end amount, and send the file segment.

[0111] The present invention saves network bandwidth and improves the efficiency of file segment transmission by sending, pulling, verifying and re-pulling message slices at the sending end and retransmitting the message when a verification error occurs.

[0112] Fig.10 A flowchart of a specific example of a Kafka-based file verification transmission method applied to a receiving end in an embodiment of the present disclosure is shown as follows: Fig.10 As shown, the Kafka-based file verification transmission method for receiving an original file provided by the receiving end in the embodiment of the present disclosure includes the following steps:

[0113] S1001, the receiver pulls the file meta information Topic;

[0114] S1002, judging whether the topic has been pulled out according to the number of topics in the meta information, if not, the process goes to S1003; if yes, the process goes to S1006;

[0115] S1003, pull the file shards;

[0116] S1004, verify whether the MD5 of the file segment content is correct. If yes, proceed to S1005; if not, proceed to S1003 and re-pull the file;

[0117] S1005, save the file segment and proceed to S1002;

[0118] S1006, combining file fragments;

[0119] S1007, verify whether the MD5 value of the combined file is the same as the MD5 value of the original file (correct), if not, then re-pull and go to S1001; if yes, then jump to S1008;

[0120] S1008, receiving is completed and the program ends.

[0121] The receiver (receiving end) of the present disclosure verifies the MD5 value of the file fragment, realizes the verification from the Kafka message middleware to the receiver, and ensures the consistency from the middleware to the receiver, thereby achieving the consistency reliability from the sender's original file to the receiver's combined file.

[0122] Fig.11 A flowchart showing another specific example of a Kafka-based file verification transmission method applied to a receiving end in an embodiment of the present disclosure is shown as follows: Fig.11 As shown, the Kafka-based file verification transmission method for pulling file fragments at the receiving end provided in the embodiment of the present disclosure includes the following steps:

[0123] S1101, the consumer pulls the message of topic Topic N in Kafka from the starting position in the metadata Topic;

[0124] S1102, storing the read message into a Buffer buffer;

[0125] S1103, verify whether the message fragment is received completely (whether the message pull offset reaches the end position in the metadata Topic). If not, the message pull offset does not reach the end position in the metadata Topic, then go to S1101 to continue pulling messages; if yes, the message offset is equal to the end position, then go to S1104;

[0126] S1104, if the Key value of the pulled messages is the same, the latest message will overwrite the previous message with the same Key value;

[0127] S1105, verify whether the message fragment has been received. If it has been received, go to S1106. If it has not been received, go to S1101 and continue to pull the message in TopicN;

[0128] S1106, combine the pulled messages.

[0129] Fig.12 A schematic diagram of message slicing and splicing in an embodiment of the present disclosure is shown. Fig.12 As shown, the message slice splicing provided in the embodiment of the present disclosure includes: slice message 121, slice message 122, slice message 1231, slice message 1232, slice message 124 and file slice 125.

[0130] Among them, slice message 121 includes a message header 1211 and a message slice 1212, slice message 122 includes a message header 1221 and a message slice 1222, slice message 1231 is an erroneous slice message including a message header 12311 and a message slice 12312, slice message 1232 is a retransmitted slice message including a message header 12321 and a message slice 12322, and slice message 124 includes a message header 1241 and a message slice 1242.

[0131] According to the message header, the message slices are spliced, and the message slices 1212, 1222, 12322, and 1242 are spliced ​​to obtain the file slice 125.

[0132] Fig.13 A schematic diagram of file segment splicing in an embodiment of the present disclosure is shown as follows: Fig.13 As shown, the file segment splicing provided in the embodiment of the present disclosure includes: file segment 131, file segment 132, file segment 133, file segment 13N and original file 134.

[0133] The file segment 13N represents the Nth file segment. The file segments 131, 132, 133 and 13N are concatenated to obtain the original file 134.

[0134] Based on the same inventive concept, the present disclosure also provides a Kafka-based file verification transmission device applied to the sending end, as described in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0135] Fig.14A schematic diagram of a Kafka-based file verification transmission device applied to a sending end in an embodiment of the present disclosure is shown. Fig.14 As shown, the device includes: a file segmentation module 141, a first sending module 142, a second sending module 143, a sending verification module 144 and a meta-information sending module 145.

[0136] Among them, the file segmentation module 141 is used to segment the original file to obtain multiple file segments; the first sending module 142 is used to send the identification information of the multiple file segments to the middleware, so that the middleware creates a one-to-one corresponding file segment topic according to the identification information of the multiple file segments; the second sending module 143 is used to divide each file segment into multiple message slices and send them to the middleware, so that the middleware stores the message slices according to the file segment topics corresponding to the message slices, wherein the message slices include the identification information of the corresponding file segment; the sending verification module 144 is used to pull each message slice in the middleware and compare and verify it with the message slice in the corresponding file segment. When the comparison and verification are consistent, it is determined that the message slice in the middleware is the first message slice, wherein the first message slice represents the message slice transmitted from the sender to the middleware and passed the verification.

[0137] In one embodiment of the present disclosure, the sending verification module 144 is also used to resend the inconsistent verification message slices to the middleware when the verification contents are inconsistent.

[0138] In one embodiment of the present disclosure, the file segmentation module 141 is further used to obtain the original file and determine the file meta information of the original file; and segment the original file according to the file meta information to obtain multiple file segments.

[0139] In one embodiment of the present disclosure, the above-mentioned Kafka-based file verification transmission device applied to the sending end also includes a metadata sending module 145, which is used to send file metadata to the middleware, so that the receiving end obtains the file metadata through the middleware for combining file fragments.

[0140] Fig.15 A schematic diagram of a Kafka-based file verification transmission device applied to a receiving end in an embodiment of the present disclosure is shown. Fig.15 As shown, the device includes: a message slice pulling module 151, a file meta-information pulling module 152 and a receiving verification module 153.

[0141] Among them, the message slice pulling module 151 is used to pull multiple first message slices in the middleware and combine them to obtain multiple file slices, wherein the first message slice represents the message slice transmitted by the sending end to the middleware and passed the verification; the file meta information pulling module 152 is used to pull file meta information, combine each file slice according to the file meta information, and determine the combined file; the receiving verification module 153 is used to verify the MD5 value of the combined file with the MD5 value of the original file. When the verification is consistent, the combined file is determined to be the original file of the sending end, and the file reception verification is completed.

[0142] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0143] Refer to the following Fig.16 1600 according to this embodiment of the present disclosure is described. Fig.16 The electronic device 1600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0144] like Fig.16 As shown, the electronic device 1600 is in the form of a general computing device. The components of the electronic device 1600 may include but are not limited to: at least one processing unit 1610, at least one storage unit 1620, and a bus 1630 connecting different system components (including the storage unit 1620 and the processing unit 1610).

[0145] The storage unit stores program codes, which can be executed by the processing unit 1610, so that the processing unit 1610 executes the steps described in the above “exemplary method” section of this specification according to various exemplary embodiments of the present disclosure.

[0146] For example, the processing unit 1610 can execute the following steps of the above-mentioned method embodiment: fragment the original file to obtain multiple file fragments; send the identification information of the multiple file fragments to the middleware, so that the middleware creates one-to-one corresponding file fragment topics according to the identification information of the multiple file fragments; divide each file fragment into multiple message slices and send them to the middleware, so that the middleware stores the message slices according to the file fragment topics corresponding to the message slices, wherein the message slices include the identification information of the corresponding file fragment; pull each message slice in the middleware and compare and verify it with the message slice in the corresponding file fragment, and when the comparison and verification are consistent, determine that the message slice in the middleware is the first message slice, wherein the first message slice represents the message slice transmitted by the sender to the middleware and passed the verification.

[0147] In one embodiment, the processing unit 1610 may execute the following steps of the above method embodiment: when the comparison verification is inconsistent, resending the verification inconsistent message slice to the middleware.

[0148] In one embodiment, the processing unit 1610 may execute the following steps of the above method embodiment: obtaining an original file, determining file meta information of the original file; and segmenting the original file according to the file meta information to obtain a plurality of file segments.

[0149] In one embodiment, the processing unit 1610 may execute the following steps of the above method embodiment: sending the file meta information to the middleware, so that the receiving end obtains the file meta information through the middleware for combining file fragments.

[0150] In one embodiment, the processing unit 1610 can execute the following steps of the above-mentioned method embodiment: pull multiple first message slices in the middleware and combine them to obtain multiple file slices, wherein the first message slice represents the message slice transmitted by the sender to the middleware and passed the verification; pull the file meta information, combine each file slice according to the file meta information, and determine the combined file; verify the MD5 value of the combined file with the MD5 value of the original file, and when the verification is consistent, determine that the combined file is the original file of the sender, and complete the file reception verification.

[0151] The storage unit 1620 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 16201 and / or a cache storage unit 16202 , and may further include a read-only storage unit (ROM) 16203 .

[0152] The storage unit 1620 may also include a program / utility 16204 having a set (at least one) of program modules 16205, such program modules 16205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0153] Bus 1630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0154] The electronic device 1600 may also communicate with one or more external devices 1640 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1600, and / or communicate with any device that enables the electronic device 1600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1650. Furthermore, the electronic device 1600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1660. As shown, the network adapter 1660 communicates with other modules of the electronic device 1600 via a bus 16630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0155] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0156] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0157] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0158] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0159] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0160] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0161] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0162] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0163] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0164] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A file verification transmission method based on Kafka, applied to the sending end, It is characterized in that include: The original file is fragmented to obtain multiple file fragments; Sending identification information of the plurality of file segments to the middleware, so that the middleware creates one-to-one corresponding file segment topics according to the identification information of the plurality of file segments; Splitting each file slice into multiple message slices and sending them to the middleware, so that the middleware stores the message slices according to the file slice topics corresponding to the message slices, wherein the message slices include identification information of the corresponding file slices; Pull each message slice in the middleware and compare and verify it with the message slice in the corresponding file slice. When the comparison and verification are consistent, determine that the message slice in the middleware is the first message slice, wherein the first message slice represents the message slice transmitted by the sender to the middleware and passed the verification.

2. The file verification transmission method based on Kafka according to claim 1, It is characterized in that Pull each message slice in the middleware and compare and verify it with the message slice in the corresponding file slice, including: When the comparison and verification are inconsistent, resend the inconsistent verification message slice to the middleware.

3. The file verification transmission method based on Kafka according to claim 1, It is characterized in that The fragmentation process of the original file includes: Obtaining the original file and determining the file meta information of the original file; The original file is segmented according to the file meta-information to obtain a plurality of file segments.

4. The file verification transmission method based on Kafka according to claim 3, It is characterized in that After obtaining the original file and determining the file meta information of the original file, the method further includes: The file meta information is sent to the middleware, so that the receiving end obtains the file meta information through the middleware for combining file fragments.

5. A file verification transmission system based on Kafka, It is characterized in that include: Sending end, middleware, receiving end; The sending end is used to slice the original file to obtain multiple file slices; send identification information of the multiple file slices to the middleware; cut each file slice into multiple message slices and send them to the middleware; pull each message slice in the middleware and compare and verify it with the message slice in the corresponding file slice, and when the comparison and verification are consistent, determine that the message slice in the middleware is the first message slice; The middleware is used to receive identification information of multiple file fragments sent by the sending end, create one-to-one corresponding file fragment topics according to the identification information of the multiple file fragments; receive multiple message slices sent by the sending end, and store the message slices according to the file fragment topics corresponding to the message slices; The receiving end is used to pull multiple first message slices in the middleware and combine them to obtain multiple file slices; pull file meta information, combine each file slice according to the file meta information, and determine the combined file; verify the MD5 value of the combined file with the MD5 value of the original file, and when the verification is consistent, determine that the combined file is the original file of the sending end, and complete the file reception verification; Among them, the message slice includes identification information of the corresponding file slice, and the first message slice represents the message slice transmitted by the sender to the middleware and passed the verification.

6. A file verification transmission device based on Kafka, applied to the sending end, It is characterized in that include: A file fragmentation module is used to fragment the original file to obtain multiple file fragments; A first sending module, used to send identification information of multiple file segments to the middleware, so that the middleware creates a one-to-one corresponding file segment topic according to the identification information of the multiple file segments; A second sending module is used to divide each file segment into multiple message segments and send them to the middleware, so that the middleware stores the message segments according to the file segment topics corresponding to the message segments, wherein the message segments include identification information of the corresponding file segments; The sending verification module is used to pull each message slice in the middleware and compare it with the message slice in the corresponding file slice. When the comparison and verification are consistent, the message slice in the middleware is determined to be the first message slice, wherein the first message slice represents the message slice transmitted by the sender to the middleware and passed the verification.

7. An electronic device, It is characterized in that include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the Kafka-based file verification and transmission method described in any one of claims 1 to 4 by executing the executable instructions.

8. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the Kafka-based file verification and transmission method described in any one of claims 1 to 4 is implemented.

Images