A data transmission method and device, electronic equipment and storage medium

Abstract

The present disclosure discloses a data transmission method and device, electronic equipment and storage medium. Based on the configuration parameters of the message queue, the first name and the first file record number of the target transmission file are obtained, and the first name is written into the first record file as the key and the first file record number is written into the first record file as the value. The second name and the second file record number of the target transmission file are obtained from the target partition, and the second name and the second file record number are written into the second record file. The second record file is read, the second name is used as the key, and the second file record number is used as the value and stored in the hash function. The first record file is read, the values of the key-value pairs in the first record file are compared with the values of the key-value pairs in the hash function, and whether the data transmission is abnormal is determined according to the comparison result. The present disclosure has simple logic and is easy to implement, has low dependence on the network and terminal, has high detection efficiency, can improve the accuracy of data transmission, and solves the problem of data loss in the data transmission process.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data transmission, and particularly relates to a data transmission auditing method and device, an electronic device and a storage medium. BACKGROUND

[0002] With the development of computer technology, more and more enterprises use message queues (MQ) as the core way of data transmission within enterprise IT systems. It has a series of functions such as low coupling, reliable delivery, broadcast, flow control, eventual consistency, etc. The currently used message queues include RabbitMQ, RocketMQ, ActiveMQ, Kafka, ZeroMQ, MetaMQ, etc.

[0003] The data transmission technology of the related art usually determines the accuracy of data transmission and whether the data transmission is successful by the feedback message of the receiving party in the process of data transmission. Taking the Kafka data transmission as an example: when Kafka transmits data, the producer receives the successful response of the Kafka server when the data is transmitted to the leader node and synchronized to all replicas. When the producer sends data to the server, if a temporary error occurs and the data cannot be sent, the producer will resend the data, thereby ensuring the integrity of the data.

[0004] Although the related message queue technology makes a relatively reliable guarantee in terms of data loss, data loss may still occur in some special scenarios. For example, taking the Kafka data transmission as an example, when the following situations occur, data loss is likely to occur:

[0005] 1. Component flow problem: data loss occurs in the upstream or downstream of Kafka.

[0006] 2. Network problem: the Kafka rate exceeds the network bandwidth. Once a sending failure occurs and there is no retry mechanism, data loss will occur and cannot be recovered.

[0007] 3. Data size problem: long single-batch data size may cause data loss. The data size produced by the producer is greater than the maximum message size configured by the consumer for pulling, which far exceeds the pulling capacity of the consumer. This large data will fail to be consumed, and may even cause Kafka to crash, thereby causing data loss.

[0008] 4. Hard disk problem: the Kafka message queue is stored in the form of log files. When the host disk fails to recover data, data loss may occur.

[0009] In the above scenarios, not only is data loss likely to occur, but the failure to detect and retransmit the lost data in a timely manner may also lead to the inability to recover the lost data. Summary of the Invention

[0010] This disclosure provides a method, apparatus, electronic device, and storage medium for auditing data transmission. Its main purpose is to address the problem that data loss is prone to occur during data transmission, and that the loss may be irretrievable due to the failure to detect and promptly retransmit the lost data.

[0011] According to a first aspect of this disclosure, a data transmission method is provided, comprising:

[0012] Based on the configuration parameters of the message queue, the first name and the first file record count of the target transmission file are obtained, and the first name is used as the key and the first file record count is used as the value to write to the first record file. The configuration parameters also include the target partition of the target transmission file. The message queue is used to store the target transmission file in the target partition, and the first file record count is used to record the number of times the target transmission file is sent.

[0013] Obtain the second name and second file record count of the target transmission file from the target partition, and write the second name and second file record count into a second record file, wherein the second file record count is used to record the number of times the target transmission file has been received;

[0014] The second record file is read, and the second name is used as the key and the number of records in the second file is used as the value. The hash function stores the data in key-value pairs.

[0015] Read the first record file, compare the value of the key-value pair in the first record file with the value of the key-value pair in the hash function, and determine whether there is an anomaly in the data transmission based on the comparison result.

[0016] Optionally, before obtaining the first name and first file record count of the target transfer file based on the message queue configuration parameters, the method further includes:

[0017] The file to be transferred is moved from the input directory to a temporary directory using a contention-based mode and added to the message queue;

[0018] Read the file name and file content of the file to be transferred from the message queue, and determine the size, path, temporary directory and backup directory of the file to be transferred as file information;

[0019] The file to be transmitted is packaged and encapsulated using the file name corresponding to the file to be transmitted as the key, and the file content and file information corresponding to the file to be transmitted as the value, to obtain the target transmission file.

[0020] Optionally, after obtaining the target transfer file, the method further includes:

[0021] Based on the target partition in the configuration parameters, the target transfer file is sent to the target partition, and the target number of target transfer files sent to the target partition is determined.

[0022] Optionally, the step of configuring the input parameters based on the message queue to obtain the first name and the first file record count of the target transmission file, and writing the first name as the key and the first file record count as the value into the first record file, includes:

[0023] The target quantity is determined as the first file record number of the target transmission file;

[0024] Write the first record file with the first name as the key and the first file record number as the value.

[0025] Upon receiving the recording signal of the first record file, it is determined that the transmission of the target file has been completed. The recording signal is used to determine that the writing of the first name and the number of records of the first file has been completed.

[0026] Optionally, the step of reading the first record file, comparing the values ​​of key-value pairs in the first record file with the values ​​of key-value pairs in the hash function, and determining whether there is an anomaly in the data transmission based on the comparison result includes:

[0027] Traverse the first record file, search for keys in the hash function that are the same as those in the first record file, and determine the keys that are the same in the hash function as the target keys;

[0028] After determining the target key, it is determined whether the value of the target key is consistent with the value in the first record file;

[0029] Based on the comparison between the value of the target key and the value in the first record file, it is determined whether there is an anomaly in the data transmission.

[0030] Optionally, determining whether there is an anomaly in the data transmission based on the comparison results includes:

[0031] If it is determined that the key in the first record file does not exist in the hash function, then the key in the first record file is determined to be an abnormal key, and the number of records in the second file corresponding to the abnormal key is determined to be 0;

[0032] If the value of the target key is determined to be consistent with the value in the first record file, then the data transmission is determined to be normal.

[0033] If the value of the target key is determined to be inconsistent with the value in the first record file, then a data transmission anomaly is determined.

[0034] The target file corresponding to the abnormal key and the target key with the abnormal data transmission is identified as an abnormal file;

[0035] Output the first name, the number of records in the first file, and the number of records in the second file corresponding to the abnormal file.

[0036] Optionally, the method further includes:

[0037] After identifying the abnormal file, obtain the backup file corresponding to the abnormal file and retransmit the backup file.

[0038] According to a second aspect of this disclosure, a data transmission apparatus is provided, comprising:

[0039] The first writing unit is used to obtain the first name and the first file record number of the target transmission file based on the configuration parameters of the message queue, and write the first name as the key and the first file record number as the value into the first record file. The configuration parameters also include the target partition of the target transmission file. The message queue is used to store the target transmission file in the target partition. The first file record number is used to record the number of times the target transmission file is sent.

[0040] The second writing unit is used to obtain the second name and the second file record count of the target transfer file from the target partition, and write the second name and the second file record count into the second record file, wherein the second file record count is used to record the number of times the target transfer file has been received;

[0041] The storage unit is used to read the second record file, use the second name as the key and the number of records in the second file as the value, and store the data in a hash function, wherein the hash function stores data in key-value pairs.

[0042] The comparison unit is used to read the first record file, compare the value of the key-value pair in the first record file with the value of the key-value pair in the hash function, and determine whether there is an anomaly in the data transmission based on the comparison result.

[0043] Optionally, the device further includes a packaging unit, the packaging unit comprising:

[0044] An add module is used to move the file to be transferred from the input directory to a temporary directory and add it to the message queue before obtaining the first name and first file record number of the target file to be transferred based on the configuration parameters of the message queue;

[0045] The reading module is used to read the file name and file content of the file to be transferred from the message queue, and to determine the size, path, temporary directory and backup directory of the file to be transferred as file information;

[0046] The encapsulation module is used to encapsulate and package the file to be transmitted by using the file name corresponding to the file to be transmitted as the key and concatenating the file content and file information corresponding to the file to be transmitted as the value, so as to obtain the target transmission file.

[0047] Optionally, the apparatus further includes a transmitting unit, the transmitting unit being used for:

[0048] After obtaining the target transfer file, based on the target partition in the configuration parameters, the target transfer file is sent to the target partition, and the target number of target transfer files sent to the target partition is determined.

[0049] Optionally, the first writing unit includes:

[0050] The first determining module is used to determine the target quantity as the first file record number of the target transmission file;

[0051] The writing module is used to write the first record file using the first name as the key and the first file record number as the value.

[0052] The second determining module is used to determine that the transmission of the target file is completed after receiving the recording signal of the first record file. The recording signal is used to determine that the writing of the first name and the number of records of the first file is completed.

[0053] Optionally, the comparison unit includes:

[0054] The search module is used to traverse the first record file, search for keys that are the same as those in the first record file in the hash function, and determine the keys that are the same as those in the hash function as the target keys;

[0055] The judgment module is used to determine, after determining the target key, whether the value of the target key is consistent with the value in the first record file;

[0056] The third determining module is used to determine whether there is an anomaly in the data transmission based on the comparison result between the value of the target key and the value in the first record file.

[0057] Optionally, the comparison unit further includes:

[0058] The fourth determining module is used to determine the key in the first record file as an abnormal key and determine the number of second file records corresponding to the abnormal key as 0 if it is determined that the key in the first record file does not exist in the hash function.

[0059] The fifth determining module is used to determine that data transmission is normal if the value of the target key is consistent with the value in the first record file.

[0060] The sixth determining module is used to determine a data transmission anomaly if the value of the target key is inconsistent with the value in the first record file.

[0061] The seventh determining module is used to determine the target transmission file corresponding to the abnormal key and the target key with abnormal data transmission as an abnormal file;

[0062] The output module is used to output the first name, the number of records in the first file, and the number of records in the second file corresponding to the abnormal file.

[0063] Optionally, the apparatus further includes a retransmission unit for:

[0064] After identifying the abnormal file, obtain the backup file corresponding to the abnormal file and retransmit the backup file.

[0065] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0066] At least one processor; and

[0067] A memory communicatively connected to the at least one processor; wherein,

[0068] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect above.

[0069] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the first aspect above.

[0070] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0071] The data transmission method, apparatus, electronic device, and storage medium disclosed herein, based on message queue configuration parameters, obtain a first name and a first file record count of a target transmission file, and write the first name as the key and the first file record count as the value into a first record file. The configuration parameters also include a target partition of the target transmission file. The message queue is used to store the target transmission file in the target partition, and the first file record count is used to record the number of times the target transmission file is sent. A second name and a second file record count of the target transmission file are obtained from the target partition, and written into a second record file. The second file record count is used to record the number of times the target transmission file is received. The second record file is read, and the second name as the key and the second file record count as the value are stored in a hash function, where the hash function stores key-value pairs. The first record file is read, and the values ​​of the key-value pairs in the first record file are compared with the values ​​of the key-value pairs in the hash function. Based on the comparison result, it is determined whether there is an anomaly in the data transmission. The data transmission method provided in this disclosure records the number of file records before and after data transmission using a first record file and a second record file. The comparison is performed by comparing key-value pairs. The logic is simple and easy to implement, with low dependence on network and terminal, high detection efficiency, and the ability to promptly detect lost data during data transmission and re-upload it, improving the accuracy of data transmission and solving the problem of data loss during data transmission. It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0072] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0073] Figure 1 A schematic flowchart illustrating a data transmission method provided in an embodiment of this disclosure;

[0074] Figure 2 A schematic diagram illustrating the detection of a data transmission method provided in an embodiment of this disclosure;

[0075] Figure 3 A schematic flowchart illustrating another data transmission method provided in an embodiment of this disclosure;

[0076] Figure 4 A schematic flowchart illustrating another data transmission method provided in an embodiment of this disclosure;

[0077] Figure 5 This is a schematic diagram of the structure of a data transmission device provided in an embodiment of the present disclosure;

[0078] Figure 6 A schematic diagram of another data transmission device provided in an embodiment of this disclosure;

[0079] Figure 7 A schematic block diagram of an example electronic device 400 provided for embodiments of this disclosure. Detailed Implementation

[0080] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0081] The following description, with reference to the accompanying drawings, outlines a data transmission method, apparatus, electronic device, and storage medium according to embodiments of the present disclosure.

[0082] Figure 1 This is a schematic flowchart illustrating a data transmission method provided in an embodiment of the present disclosure.

[0083] like Figure 1 As shown, the method includes the following steps:

[0084] Step 101: Based on the configuration parameters of the message queue, obtain the first name and the first file record count of the target transmission file, and write the first name as the key and the first file record count as the value into the first record file. The configuration parameters also include the target partition of the target transmission file. The message queue is used to store the target transmission file in the target partition, and the first file record count is used to record the number of times the target transmission file is sent.

[0085] In this embodiment, the message queue is a middleware with a queue structure. Data placed into this middleware can be returned directly without immediate system processing. Instead, a separate program reads this data and processes it sequentially. Message queues primarily address issues such as application coupling, asynchronous processing, and traffic shaping. Currently, commonly used message queues include RabbitMQ, RocketMQ, ActiveMQ, Kafka, ZeroMQ, and MetaMQ. Some databases, such as Redis, MySQL, and phxsql, can also implement message queue functionality.

[0086] To detect whether data transmission is normal, the first name is used as the key and the first file record count is used as the value, written to a first log file for comparison to determine whether data transmission was successful. The message queue configuration parameters include the first name of the target file, the first file record count, and the target partition. The first name and the first file record count are used to verify data transmission. The first file record count includes the number of parsing errors, successful transmissions, failed transmissions, and transmissions to the target partition. The first log file is a file with the .stat extension, used to record the first name and the first file record count in key-value pairs as the basis for data transmission detection. The first log file also records other information about the target file, such as its size, storage path, and storage time.

[0087] To better understand the first record file, this application provides an example of the file content of the first record file, for example:

[0088] The first record in the file named in_stat_20221020 contains the following content:

[0089] Fi leName:gprs_ol_30200_B_D08_452718909.txt;FileSize:2562235;Earlist:20221023;latest:20221023;ExtALines:0;Total Lines:1512;SendLines:1512;NonTimeoutExceptionLines:0;TimeoutExceptionLines:0;StartTime:20221023112804;EndTim e:20221023112805;Speed:2119;TopicList:{CloudXdrTopicA0.1:1512;};BillDateList:{20221023:1512;};TargetLines:0

[0090] From the first record file in the example above, information related to the target transfer file can be obtained. For example, FileName:gprs_ol_30200_B_D08_452718909.txt is the first name of the target transfer file, FileSize:2562235 is the size of the target transfer file, TotalLines:1512 is the total number of the target transfer files, and SendLines:1512 is the number of target transfer files sent to the target partition, etc.

[0091] Step 102: Obtain the second name and second file record count of the target transmission file from the target partition, and write the second name and second file record count into a second record file. The second file record count is used to record the number of times the target transmission file has been received.

[0092] To verify whether data loss has occurred during data transmission, after acquiring the target transmission file, the second name and second file record count of the target transmission file are written into a second record file. The second file record count is used to record the number of times the target transmission file has been received. The second record file is used to record the second name, second file record count, and related information of the received target transmission file, so as to compare it with the first record file and find the target transmission file where data loss has occurred.

[0093] Step 103: Read the second record file, use the second name as the key and the number of records in the second file as the value, and store it in a hash function. The hash function stores data in key-value pairs.

[0094] To improve search efficiency, this invention employs a hash comparison method from the Perl language. Perl is a feature-rich computer programming language that runs on over 100 computer platforms and is widely applicable. Originally developed for text processing, it is now used for various tasks, including system administration, web development, network programming, and GUI development. In Perl, hash data structures can be used to store key-value pairs. A hash function is a mapping relationship that calculates the storage location of an element based on its key. Hash temporarily stores data in memory as key-value pairs. The previously written hash key-value pair (in this application, the key-value pair in the first record file) is directly retrieved using the key (which can be a number or a character, similar to an index). By comparing the previously written value with the current value using the key, the comparison of two sets of values ​​can be efficiently achieved, improving search efficiency.

[0095] Step 104: Read the first record file, compare the value of the key-value pair in the first record file with the value of the key-value pair in the hash function, and determine whether there is an anomaly in the data transmission based on the comparison result.

[0096] In this embodiment, after storing the second record file in a hash function, the first record file is read, and the first record file is traversed line by line. A key identical to the one in the first record file is searched in the hash function. If it is determined that the key in the first record file does not exist in the hash function, the key in the first record file is identified as an abnormal key, and the number of second file records corresponding to the abnormal key is set to 0. If it is determined that the key in the first record file exists in the hash function, the identical key in the hash function is identified as the target key. After identifying the target key, it is determined whether the value of the target key is consistent with the value in the first record file. If the value of the target key is consistent with the value in the first record file, the data transmission is determined to be normal. If the value of the target key is inconsistent with the value in the first record file, the data transmission is determined to be abnormal. The target transmission file corresponding to the abnormal key and the target key of the abnormal data transmission is identified as an abnormal file. The first name, the number of first file records, and the number of second file records corresponding to the abnormal file are output.

[0097] To better understand the comparison between the first record file and the second record file, embodiments of this application also provide a schematic diagram of a data transmission method detection, such as... Figure 2 As shown, the first record file and the second record file are read first, and then the same key is searched in the two record files. After the same key is identified, the value corresponding to the key is compared. If the values ​​are the same, the data transmission ends. If they are not the same, the relevant information of the abnormal file is output. The relevant information includes the first name of the abnormal file, the number of records in the first file and the number of records in the second file, so as to determine the data lost by the abnormal file during the transmission. After outputting the relevant information of the abnormal file, the data transmission ends.

[0098] By generating a first record file before data transmission to record the first file record number and other relevant information of the target file, and generating a second record file after data transmission to record the second transmission information of the target file, the data transmission is detected only by comparing the first and second record files, without detecting the data transmission process. The logic is simple and easy to implement, and efficient detection is achieved through hash comparison, quickly finding abnormal files in the data transmission.

[0099] The data transmission method provided in this disclosure, based on the configuration parameters of a message queue, obtains a first name and a first file record count of a target transmission file, and writes the first name as the key and the first file record count as the value into a first record file. The configuration parameters also include a target partition of the target transmission file. The message queue is used to store the target transmission file in the target partition, and the first file record count is used to record the number of times the target transmission file is sent. A second name and a second file record count of the target transmission file are obtained from the target partition, and written into a second record file. The second file record count is used to record the number of times the target transmission file is received. The second record file is read, and the second name as the key and the second file record count as the value are stored in a hash function, where the hash function stores key-value pairs. The first record file is read, and the values ​​of the key-value pairs in the first record file are compared with the values ​​of the key-value pairs in the hash function. Based on the comparison result, it is determined whether there is an anomaly in the data transmission. The data transmission method provided in this disclosure, compared with other related technologies that determine the accuracy and success of data transmission by relying on messages from the receiver during the data transmission process, uses a first record file and a second record file to record the number of file records before and after the data transmission, and compares the values ​​of key-value pairs. The logic is simple and easy to implement, with low dependence on network and terminal, high detection efficiency, and the ability to promptly detect lost data during the data transmission process and re-upload it, thereby improving the accuracy of data transmission and solving the problem of data loss during the data transmission process.

[0100] In the embodiments provided in this application, the data transmission method further includes a data transmission encapsulation process and a transmission process. To better understand the encapsulation process in the data transmission method, Figure 3 This is a schematic flowchart illustrating another data transmission method provided in an embodiment of this disclosure.

[0101] like Figure 3 As shown, the method also includes the following steps:

[0102] Step 201: Using a contention mode, the file to be transferred is moved from the input directory to a temporary directory and added to the message queue.

[0103] The contention mode, also known as the Work mode, involves producers in a message queue sending messages to an exchange. The exchange then forwards the messages to the message queue based on their attributes. Multiple consumers simultaneously bind to and listen to a single message queue, creating a contention for messages.

[0104] Step 202: Read the file name and file content of the file to be transferred from the message queue, and determine the size, path, temporary directory and backup directory of the file to be transferred as file information.

[0105] File encapsulation refers to combining related resources such as software, files, or data together and packaging them into a single file or archive. This process merges multiple files or resources into a single entity to facilitate distribution, transmission, or deployment. To facilitate the transmission of the target file and to obtain information related to the target file being written to a first record file, file information is retrieved from the message queue.

[0106] Step 203: Using the file name corresponding to the file to be transmitted as the key, and concatenating the file content and file information corresponding to the file to be transmitted as the value, the file to be transmitted is packaged and packaged to obtain the target transmission file.

[0107] To obtain the target file for transfer, a file of type `FileInfo` is encapsulated using a concatenation of file content and file information as the value and the filename as the key. The `FileInfo` class, located in the `System.IO` namespace, provides a series of properties and methods for manipulating files. Through the `FileInfo` class, we can easily obtain information about the file, such as file size, creation date, and modification date. Furthermore, the `FileInfo` class provides convenient methods for file copying, moving, and deletion. For example, in a Kafka cluster, the Producer Record component can be used to encapsulate the file to be transferred into a target file such as `ProducerRecord(CloudXdrTopicA0.1, filename, text@info)`.

[0108] The above encapsulation method allows for the simultaneous acquisition of relevant information written to the first record file during the encapsulation of the file to be transmitted. Furthermore, the encapsulation of the file to be transmitted facilitates the subsequent transmission of the target file.

[0109] In some embodiments, after encapsulating the file to be transmitted and obtaining the target transmission file, a load balancing method is used to send the encapsulated target transmission file (the FileInfo class file in the above embodiments) to different partitions of different topics in the message middleware (the Kafka cluster in the above embodiments). Simultaneously, a target number of target transmission files to be sent to the target partitions is determined. Load balancing (LB) is an essential component of high-concurrency, high-availability systems, aiming to distribute network traffic evenly across multiple servers to improve the overall system response speed and availability. For example, if there are n partitions and n target transmission files, each target transmission file is sent to a different partition to achieve load balancing.

[0110] In some embodiments, after the target transfer file is sent to the target partition, the system waits for a recording signal from the first record file. Upon receiving the recording signal from the first record file, the system determines that the transmission of the target transfer file is complete. Otherwise, it waits until the recording signal is received before continuing to send the next target transfer file. The recording signal is used to determine the successful writing of the first name and the number of records in the first file to avoid omissions.

[0111] To better understand the data transmission process, taking a Kafka cluster as an example, this application also provides another data transmission flow diagram, such as... Figure 4 As shown, when the file to be transferred is input, the encapsulation and backup of the file to be transferred are completed, and the first record file is generated at the same time. When the target file to be transferred is output, the corresponding second file to be transferred is generated.

[0112] In some embodiments, after identifying the abnormal file by comparing the first record file and the second record file, a backup file corresponding to the abnormal file is obtained and the backup file is retransmitted to improve the accuracy of data transmission and avoid data loss.

[0113] By encapsulating the file to be transmitted and transmitting a backup of the abnormal file after identifying the abnormal file, the accuracy of data transmission is improved and the probability of data loss is reduced.

[0114] In summary, the embodiments disclosed herein achieve the following effects:

[0115] 1. By generating a first record file before data transmission to record the first file record number and other relevant information of the target file, and generating a second record file after data transmission to record the second transmission information of the target file, data transmission is detected only by comparing the first record file and the second record file, without detecting the data transmission process. The logic is simple and easy to implement, and efficient detection is achieved through hash comparison method, quickly finding abnormal files in data transmission.

[0116] 2. By encapsulating the file to be transmitted and transmitting a backup of the abnormal file after identifying the abnormal file, the accuracy of data transmission is improved and the probability of data loss is reduced.

[0117] Corresponding to the data transmission method described above, the present invention also proposes a data transmission device. Since the device embodiments of the present invention correspond to the method embodiments described above, details not disclosed in the device embodiments can be referred to in the method embodiments described above, and will not be repeated here.

[0118] Figure 5 This is a schematic diagram of the structure of a data transmission device provided in an embodiment of the present disclosure, as shown below. Figure 5 As shown, it includes: a first writing unit 31, a second writing unit 32, a storage unit 33, and a comparison unit 34.

[0119] The first writing unit 31 is used to obtain the first name and the first file record number of the target transmission file based on the configuration parameters of the message queue, and write the first name as the key and the first file record number as the value into the first record file. The configuration parameters also include the target partition of the target transmission file. The message queue is used to store the target transmission file in the target partition. The first file record number is used to record the number of times the target transmission file is sent.

[0120] The second writing unit 32 is used to obtain the second name and the second file record count of the target transfer file from the target partition, and write the second name and the second file record count into the second record file, wherein the second file record count is used to record the number of times the target transfer file has been received;

[0121] Storage unit 33 is used to read the second record file, use the second name as the key and the number of records in the second file as the value, and store it in a hash function, wherein the hash function stores in key-value pairs.

[0122] The comparison unit 34 is used to read the first record file, compare the value of the key-value pair in the first record file with the value of the key-value pair in the hash function, and determine whether there is an anomaly in the data transmission based on the comparison result.

[0123] The data transmission method and apparatus provided in this disclosure, based on the configuration parameters of a message queue, obtains a first name and a first file record count of a target transmission file, and writes the first name as the key and the first file record count as the value into a first record file. The configuration parameters also include a target partition of the target transmission file. The message queue is used to store the target transmission file in the target partition, and the first file record count is used to record the number of times the target transmission file is sent. A second name and a second file record count of the target transmission file are obtained from the target partition, and written into a second record file. The second file record count is used to record the number of times the target transmission file is received. The second record file is read, and the second name as the key and the second file record count as the value are stored in a hash function, where the hash function stores key-value pairs. The first record file is read, and the values ​​of the key-value pairs in the first record file are compared with the values ​​of the key-value pairs in the hash function. Based on the comparison result, it is determined whether there is an anomaly in the data transmission. The data transmission method provided in this disclosure, compared with other related technologies that determine the accuracy and success of data transmission by relying on feedback messages from the receiver during the data transmission process, uses a first record file and a second record file to record the number of file records before and after data transmission, and compares the values ​​of key-value pairs. This method is logically simple and easy to implement, has low dependence on networks and terminals, high detection efficiency, and can promptly detect lost data during data transmission and re-upload it, improving the accuracy of data transmission and solving the problem of data loss during data transmission. Furthermore, in one possible implementation of this embodiment, such as... Figure 6 As shown, the device further includes a packaging unit 35, the packaging unit comprising:

[0124] Add module 351, which is used to transfer the file to be transferred from the input directory to the temporary directory and add it to the message queue before obtaining the first name and first file record number of the target file to be transferred based on the configuration parameters of the message queue;

[0125] The reading module 352 is used to read the file name and file content of the file to be transferred from the message queue, and to determine the size, path, temporary directory and backup directory of the file to be transferred as file information;

[0126] The encapsulation module 353 is used to encapsulate and package the file to be transmitted by using the file name corresponding to the file to be transmitted as the key and concatenating the file content and file information corresponding to the file to be transmitted as the value, so as to obtain the target transmission file.

[0127] Furthermore, in one possible implementation of this embodiment, such as Figure 6 As shown, the device further includes a transmitting unit, the transmitting unit 36 ​​being used for:

[0128] After obtaining the target transfer file, based on the target partition in the configuration parameters, the target transfer file is sent to the target partition, and the target number of target transfer files sent to the target partition is determined.

[0129] Furthermore, in one possible implementation of this embodiment, such as Figure 6 As shown, the first writing unit 31 includes:

[0130] The first determining module 311 is used to determine the target quantity as the first file record number of the target transmission file;

[0131] The writing module 312 is used to write the first record file with the first name as the key and the first file record number as the value.

[0132] The second determining module 313 is used to determine that the transmission of the target file is completed after receiving the recording signal of the first record file. The recording signal is used to determine that the writing of the first name and the number of records of the first file is completed.

[0133] Furthermore, in one possible implementation of this embodiment, such as Figure 6 As shown, the comparison unit 34 includes:

[0134] The lookup module 341 is used to traverse the first record file, search for the same key in the hash function as the first record file, and determine the same key in the hash function as the target key;

[0135] The judgment module 342 is used to determine, after determining the target key, whether the value of the target key is consistent with the value in the first record file;

[0136] The third determining module 343 is used to determine whether there is an anomaly in the data transmission based on the comparison result between the value of the target key and the value in the first record file.

[0137] Furthermore, in one possible implementation of this embodiment, such as Figure 5 As shown, the comparison unit 34 further includes:

[0138] The fourth determining module 344 is used to determine the key in the first record file as an abnormal key and determine the number of second file records corresponding to the abnormal key as 0 if it is determined that the key in the first record file does not exist in the hash function.

[0139] The fifth determining module 345 is used to determine that data transmission is normal if the value of the target key is consistent with the value in the first record file.

[0140] The sixth determining module 346 is used to determine that the data transmission is abnormal if the value of the target key is inconsistent with the value in the first record file.

[0141] The seventh determining module 347 is used to determine the target transmission file corresponding to the abnormal key and the target key of the data transmission abnormality as an abnormal file;

[0142] The output module 348 is used to output the first name, the number of records in the first file, and the number of records in the second file corresponding to the abnormal file.

[0143] Furthermore, in one possible implementation of this embodiment, such as Figure 6 As shown, the device further includes a retransmission unit 37, used for:

[0144] After identifying the abnormal file, obtain the backup file corresponding to the abnormal file and retransmit the backup file.

[0145] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of this embodiment, and the principle is the same, so it is not limited in this embodiment.

[0146] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0147] Figure 7 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0148] like Figure 7As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 402 or a computer program loaded from storage unit 408 into RAM (Random Access Memory) 403. RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. I / O (Input / Output) interface 405 is also connected to bus 404.

[0149] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0150] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as data transfer methods. For example, in some embodiments, the data transfer method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the aforementioned data transmission method by any other suitable means (e.g., by means of firmware).

[0151] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0153] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0154] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0155] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0156] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0157] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0158] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0159] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data transmission method, characterized in that, include: Based on the configuration parameters of the message queue, the first name and the first file record count of the target transmission file are obtained, and the first name is used as the key and the first file record count is used as the value to write to the first record file. The configuration parameters also include the target partition of the target transmission file. The message queue is used to store the target transmission file in the target partition, and the first file record count is used to record the number of times the target transmission file is sent. Obtain the second name and second file record count of the target transfer file from the target partition, and write the second name and second file record count into a second record file, wherein the second file record count is used to record the number of times the target transfer file has been received; The second record file is read, and the second name is used as the key and the number of records in the second file is used as the value. The hash function stores the data in key-value pairs. Read the first record file, compare the value of the key-value pair in the first record file with the value of the key-value pair in the hash function, and determine whether there is an anomaly in the data transmission based on the comparison result.

2. The method according to claim 1, characterized in that, Before obtaining the first name and first file record count of the target file based on the message queue configuration parameters, the method further includes: The file to be transferred is moved from the input directory to a temporary directory using a contention-based mode and added to the message queue; Read the file name and file content of the file to be transferred from the message queue, and determine the size, path, temporary directory and backup directory of the file to be transferred as file information; The file to be transmitted is packaged and encapsulated using the file name corresponding to the file to be transmitted as the key, and the file content and file information corresponding to the file to be transmitted as the value, to obtain the target transmission file.

3. The method according to claim 2, characterized in that, After obtaining the target transfer file, the method further includes: Based on the target partition in the configuration parameters, the target transfer file is sent to the target partition, and the target number of target transfer files sent to the target partition is determined.

4. The method according to claim 3, characterized in that, The message queue-based input configuration parameters obtain the first name and first file record count of the target transmission file, and write the first name as the key and the first file record count as the value into the first record file, including: The target quantity is determined as the first file record number of the target transmission file; Write the first record file with the first name as the key and the first file record number as the value. Upon receiving the recording signal of the first record file, it is determined that the transmission of the target file has been completed. The recording signal is used to determine that the writing of the first name and the number of records of the first file has been completed.

5. The method according to claim 1, characterized in that, The step of reading the first record file, comparing the values ​​of key-value pairs in the first record file with the values ​​of key-value pairs in the hash function, and determining whether there is an anomaly in the data transmission based on the comparison result includes: Traverse the first record file, search for keys in the hash function that are the same as those in the first record file, and determine the keys that are the same in the hash function as the target keys; After determining the target key, it is determined whether the value of the target key is consistent with the value in the first record file; Based on the comparison between the value of the target key and the value in the first record file, it is determined whether there is an anomaly in the data transmission.

6. The method according to claim 5, characterized in that, The step of determining whether there is an anomaly in data transmission based on the comparison results includes: If it is determined that the key in the first record file does not exist in the hash function, then the key in the first record file is determined to be an abnormal key, and the number of records in the second file corresponding to the abnormal key is determined to be 0; If the value of the target key is determined to be consistent with the value in the first record file, then the data transmission is determined to be normal. If the value of the target key is determined to be inconsistent with the value in the first record file, then a data transmission anomaly is determined. The target file corresponding to the abnormal key and the target key with the abnormal data transmission is identified as an abnormal file; Output the first name, the number of records in the first file, and the number of records in the second file corresponding to the abnormal file.

7. The method according to claim 6, characterized in that, The method further includes: After identifying the abnormal file, obtain the backup file corresponding to the abnormal file and retransmit the backup file.

8. A data transmission method, characterized in that, include: The first writing unit is used to obtain the first name and the first file record number of the target transmission file based on the configuration parameters of the message queue, and write the first name as the key and the first file record number as the value into the first record file. The configuration parameters also include the target partition of the target transmission file. The message queue is used to store the target transmission file in the target partition. The first file record number is used to record the number of times the target transmission file is sent. The second writing unit is used to obtain the second name and the second file record count of the target transfer file from the target partition, and write the second name and the second file record count into the second record file, wherein the second file record count is used to record the number of times the target transfer file has been received; The storage unit is used to read the second record file, use the second name as the key and the number of records in the second file as the value, and store the data in a hash function, wherein the hash function stores data in key-value pairs. The comparison unit is used to read the first record file, compare the value of the key-value pair in the first record file with the value of the key-value pair in the hash function, and determine whether there is an anomaly in the data transmission based on the comparison result.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images