Safe and efficient data transmission method and device, equipment and storage medium

Abstract

The invention provides a safe and efficient data transmission method and device, equipment and a storage medium. Comprising the steps that a database is generated, the database comprises one or more data tables, and the data tables are used for storing to-be-transmitted data; defining a thread pool, wherein the thread pool comprises a plurality of threads; generating a blank file and a record table, reading the to-be-transmitted data from the database by each thread according to the size of the blank file, and writing the to-be-transmitted data into the blank file to obtain a to-be-compressed file; the record table is used for recording the name of a to-be-compressed file, the size of the to-be-compressed file, the name of a data table and the position of to-be-transmitted data in the table; compressing the to-be-compressed file to obtain a compressed file; encrypting the compressed file to obtain an encrypted file; and transmitting the encrypted file and the record table to a receiver.

Description

Technical Field

[0001] The present invention belongs to the field of data transmission, and in particular relates to a safe and efficient data transmission method and device, equipment and storage medium. Background Art

[0002] With the rapid development of Internet technology, secure data transmission has become an important issue in the information age. Whether it is the privacy protection of individual users or the security of core assets of enterprises and institutions, they all rely on efficient and reliable data encryption transmission technology.

[0003] The traditional file transfer process generally involves reading all the data first, then compressing and encrypting it, and finally transmitting it. If there is a problem in any step in the middle, it will affect subsequent processing, which will result in low transmission efficiency.

[0004] How to develop a new encrypted transmission method that can ensure the security of data transmission while taking into account efficiency has become a research hotspot in the current information security field. Summary of the invention

[0005] The present invention provides a safe and efficient data transmission method and device, equipment and storage medium, which can ensure the security of data and ensure the transmission efficiency of data.

[0006] On the one hand, a secure and efficient data transmission method is provided, comprising: Generate a database, the database includes one or more data tables, and the data tables are used to store the data to be transmitted; Define a thread pool, which includes multiple threads; Generate a blank file and a record table. Each thread reads the data to be transmitted from the database according to the size of the blank file, and writes the data to be transmitted into the blank file to obtain the file to be compressed. The record table is used to record the name of the file to be compressed, the size of the file to be compressed, the data table name and the position of the data to be transmitted in the table. Compress the file to be compressed to obtain a compressed file; Encrypt the compressed file to obtain an encrypted file; Transmit the encrypted file and log sheet to the recipient.

[0007] Optionally, each thread reads data from the database according to the size of the blank file and writes the data into the blank file to obtain a file to be compressed, including: Determine the standard size of a single piece of data based on the type of data stored in the data table; Determine the standard number of data items that a blank file can store based on the standard size; Randomly extract multiple data to be transmitted from the data table according to the standard number of data; Based on the total size of the extracted multiple pieces of data to be transmitted, determining the deviation between the total size of the multiple pieces of files to be transmitted and the size of the blank file; Adjust the number of standard data items according to the deviation; Read data from the data table according to the standard number of data entries and write them into a blank file.

[0008] Optionally, read data from the data table according to the standard number of data entries and write it to a blank file, including: Using cursor processing, data is read from the data table according to the standard number of data entries and written into a blank file.

[0009] Optionally, the compressed file is encrypted to obtain an encrypted file, including: The digital envelope algorithm and the block encryption algorithm are used to encrypt the compressed file to obtain an encrypted file.

[0010] Optionally, the compressed file is encrypted using a digital envelope algorithm and a block encryption algorithm to obtain an encrypted file, including: The compressed file is encrypted using the session key to obtain the ciphertext; A replacement table is generated by using a shuffling algorithm, wherein the replacement table includes a plurality of bytes; According to a preset block size, the ciphertext is divided into a plurality of blocks in a first preset order; when the ciphertext data in the last block is smaller than the block size, the last block is padded with padding bytes; Replace the bytes in each block with the bytes in the replacement table and reverse the order of the bytes in the block after the replacement is completed; Connecting the multiple blocks according to a second preset order to obtain a first encrypted file; The session key is encrypted according to the public key of the receiver to obtain a second encrypted file; the first encrypted file, the second encrypted file and the replacement table constitute an encrypted file.

[0011] Optionally, the encrypted file and the record table are transmitted to the receiver through the message middleware, including: Generate a message queue and set the queue's persistence to 1; Upload the encrypted file to the message queue, and the message queue sends the encrypted file to the recipient; The receiver sends a receipt message to the sender through the message queue based on the encrypted file and record table.

[0012] Optionally, after the recipient receives the encrypted file, the decryption process of the encrypted file is as follows: The receiver decrypts the second encrypted file according to the receiver's private key to obtain a session key; According to the preset block size, the first encrypted file is divided according to the second preset order to obtain a plurality of decrypted blocks; Replace the bytes in the decrypted block according to the inverse of the substitution table and reverse the byte order after the replacement is completed; Combining the multiple decryption blocks according to a first preset order to obtain a first decrypted file; removing padding characters from the first decrypted file; Decrypting the first decryption file using the ciphertext to obtain a compressed file; Decompress the compressed file to obtain the file to be compressed; The data in the file to be compressed is rearranged according to the record table to obtain the data to be transmitted.

[0013] In another aspect, a secure and efficient data transmission device is provided, comprising: A generation module is used to generate a database, which includes one or more data tables, and the data tables are used to store data to be transmitted; define a thread pool, which includes multiple threads; generate a blank file and a record table, each thread reads the data to be transmitted from the database according to the size of the blank file, and writes the data to be transmitted into the blank file to obtain a file to be compressed; the record table is used to record the name of the file to be compressed, the size of the file to be compressed, the name of the data table and the position of the data to be transmitted in the table; The file processing module is used to compress the file to be compressed to obtain a compressed file; and encrypt the compressed file to obtain an encrypted file; The transmission module is used to transmit the encrypted file and the record table to the receiver.

[0014] On the other hand, an electronic device is provided, comprising the secure and efficient data transmission device as described above.

[0015] On the other hand, a computer-readable storage medium is provided, wherein at least one program code is stored in the computer-readable storage medium, and the program code is executed by a processor to implement the secure and efficient data transmission method as described in any one of the above items.

[0016] The technical solution provided by the embodiments of the present disclosure brings the following beneficial effects: In the disclosed embodiment, a safe and efficient data transmission method is provided, by defining multiple thread pools to provide multiple threads, and the data to be transmitted is split by multiple threads, and each thread can perform data reading, data compression, data encryption and data transmission processes, thereby doubling the overall transmission efficiency of the file to be transmitted. It can also avoid the problem that the overall transmission efficiency of the file to be transmitted is affected by abnormalities in the data reading, data compression, data encryption or data transmission process of a part of the data in the transmission file. After the transmission is completed, the recipient can decrypt the file and re-combine the compressed file through the record table to obtain the data to be transmitted.

[0017] Furthermore, when the method provided in the present application is used for data transmission, if the receiving party confirms through the record table that there is a problem with a certain part of the decrypted file to be compressed, the receiving party can directly confirm which specific file to be compressed has the problem and the location of the stored data in the database based on the record table. Thus, the sending party can modify the file to be compressed with the problem based on the record table and resend it to the receiving party, without having to reprocess all the files to be transmitted and resend them to the receiving party. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A method flow chart of a secure and efficient data transmission method provided by an embodiment of the present disclosure; Figure 2 A structural block diagram of a secure and efficient data transmission device provided in an embodiment of the present disclosure; Figure 3 A structural block diagram of an electronic device provided in an embodiment of the present disclosure.

[0020] 11: Acquisition module; 12: File processing module; 13: Transmission module; 21: processor; 22: memory. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Figure 1 A method flow chart of a secure and efficient data transmission method provided by an embodiment of the present disclosure. Figure 1 ,include: S1. Generate a database, which includes one or more data tables, and the data tables are used to store data to be transmitted.

[0023] In the disclosed embodiment, the database is used to receive data transmitted from upstream and store it as band transmission data.

[0024] When the database receives data transmitted from upstream, it will perform data source detection tasks to check the validity, authority and integrity of the data source to ensure the reliability and security of the data.

[0025] After the database is built, a data overview table can also be generated to simultaneously record the name, format, number of transmissions, transmission status and other information of the data table that needs to be transmitted in the database.

[0026] In the embodiment of the present disclosure, after the database is generated, the following operations are also performed: Data preparation status check: confirm whether the data source is available, and check the network connection and service status.

[0027] Check whether the data has been generated or updated on time and whether it meets the predetermined time window.

[0028] Metadata acquisition: Get metadata of the data source, including data structure, field type, data volume, etc. Confirm the latest update time of the data to ensure that the latest data is used.

[0029] Format validation: Check whether the data format meets expectations, such as date format, number range, string length, etc. Use regular expressions or built-in validation functions for format validation.

[0030] Data type checking: Ensure that the data type of each field is consistent with the predefined type, for example, ensure that an integer field does not contain non-numeric characters.

[0031] Business rule validation: Check the validity of data based on business logic, for example, check whether a field is empty, whether a value is within the allowed range, etc.

[0032] Authentication: Verify the identity of the user accessing the data source and ensure that the user's credentials (such as username and password, API key, etc.) are valid.

[0033] Permission check: Confirm the user's access rights to a specific data source or dataset to ensure that the user has the appropriate read / write permissions.

[0034] Logging: Record access logs, including user identity, access time, accessed data source, and other information for subsequent auditing and tracking.

[0035] S2. Define a thread pool, which includes multiple threads.

[0036] In the disclosed embodiment, the number of core threads, the maximum number of threads, the task queue, the thread idle time, the time unit, the thread factory, and the custom rejection strategy of the thread pool are determined, wherein the rejection strategy uses CallerRunsPolicy, which can ensure that the transmission task can eventually be executed.

[0037] In the embodiment of the present disclosure, when defining a thread pool, a synchronization counter (Count Down Latch) is also defined, and the initial value of the synchronization counter is equal to the thread value in the thread pool. Whenever a thread completes its task, the value of the synchronization counter is reduced by one. When the value of the synchronization counter is reduced to zero, it can be determined that all threads have completed tasks such as data reading, data compression, data encryption, and data transmission.

[0038] S3. Generate a blank file and a record table. Each thread reads the data to be transmitted from the database according to the size of the blank file, and writes the data to be transmitted into the blank file to obtain the file to be compressed. The record table is used to record the name of the file to be compressed, the size of the file to be compressed, the data table name and the position of the data to be transmitted in the table.

[0039] In one example, step S3 includes the following steps: The first step is to determine the standard size of a single piece of data based on the type of data stored in the data table.

[0040] In the disclosed embodiment, the data type stored in the data table will affect the size of a single piece of data. Different types of data will have different data sizes under the same field length. For example, numeric and character data will have different single piece of data sizes under the same field length.

[0041] The data stored in a data table are generally of the same type. Therefore, the standard size of a single piece of data in the data table can be determined based on the type of data stored in the data table.

[0042] The standard size refers to the size of a certain type of data under a certain standard field length. The standard field length can be set selectively.

[0043] Second, determine the standard number of data items that a blank file can store based on the standard size.

[0044] Third, randomly extract multiple data to be transmitted from the data table according to the standard number of data.

[0045] The multiple randomly selected data to be transmitted are used to correct the errors in the number of marked data to ensure that the size of the multiple data to be transmitted actually written into the blank file is closer to the size of the blank file. Therefore, the data to be transmitted is extracted here according to the standard number of data to ensure that it is more consistent with the actual situation.

[0046] Fourth, based on the total size of the extracted multiple pieces of data to be transmitted, determine the deviation between the total size of the multiple pieces of files to be transmitted and the size of the blank file.

[0047] Although the types of data to be transmitted stored in the data table are the same, since the field lengths of each piece of data to be transmitted may be different, the size of each piece of data to be transmitted may be different, resulting in different total sizes of the same number of pieces of data to be transmitted read before and after.

[0048] In order to reduce the error of the standard data number, multiple data to be transmitted are randomly selected to correct the error of the standard data number.

[0049] Fifth, adjust the number of standard data items according to the deviation.

[0050] For example, if the size of the multiple data to be transmitted extracted according to the standard data number is larger than the size of the blank file, the standard data number is reduced accordingly. Otherwise, the standard data number is increased. Ensure that more data to be transmitted can be stored in the blank file, and the total size of the multiple data to be transmitted extracted is not larger than the size of the blank file.

[0051] Sixth, read data from the data table according to the standard number of data entries and write it into a blank file.

[0052] In the embodiment of the present disclosure, when reading data, the following method steps are adopted: Using cursor processing, data is read from the data table according to the standard number of data entries and written into a blank file.

[0053] In the disclosed embodiment, the cursor processing method first sets multiple cursors in the data table according to the number of standard data items, divides the data to be transmitted in the data table into multiple parts, and then reads and writes the data to be transmitted directly according to the position of the cursor. Through cursor processing, it is possible to avoid different threads reading the same data to be transmitted when multiple threads read and write data.

[0054] In the embodiment of the present disclosure, when writing data, an AtomicInteger object is also defined to ensure that the number of data items will not be counted repeatedly in a multi-threaded environment, a File object is defined to package the data and store it on the disk, and a BufferedWriter is defined to write text to a character output stream, while providing a buffering function to improve writing efficiency.

[0055] In the embodiment of the present disclosure, through the above method, a large amount of data to be transmitted can be split into multiple threads for processing, thereby ensuring the transmission efficiency of the data to be transmitted.

[0056] In the disclosed embodiment, each thread reads data from the database. When reading, the state of the data table in the data overview table is first determined (for example, whether the data table has been transferred, which data in the data table is being transferred, and which has not been transferred). Then, the location of the data to be transferred in the data table is determined and read. When reading, the state of the currently read data is updated in the data overview table.

[0057] In the disclosed embodiment, only when the number of data table transmissions recorded in the data overview table is less than or equal to 3, and the transmission status is a record of not started or failed, can the thread read data from the data table. Data tables that do not meet the above conditions will be recorded in the user interface, and through log records and problem analysis, the transmission status will be set to not started after the problem is solved to meet the conditions and be re-scanned.

[0058] The number of transmissions of the data table is initially set to 0. Only when all the data in the data table has gone through the process of data reading, data compression, data encryption and data transmission, the number of transmissions will be increased by 1.

[0059] If the transmission status of the data table is "Not Started", it means that all the data to be transmitted in the data table has not been completely transmitted.

[0060] If the transmission status in the data table is in a failed state, it means that when the data table is reading data, compressing data, encrypting data, and transmitting data, a certain process encounters an obstacle, resulting in the inability to transmit data.

[0061] S4. Compress the file to be compressed to obtain a compressed file.

[0062] In an example provided in the present disclosure, a file to be compressed is compressed using a GZIP algorithm to obtain a compressed file.

[0063] S5. Encrypt the compressed file to obtain an encrypted file.

[0064] In the embodiment of the present disclosure, step S5 includes: The digital envelope algorithm and the block encryption algorithm are used to encrypt the compressed file to obtain an encrypted file.

[0065] In the disclosed embodiment, a mixture of multiple encryption algorithms is used for encryption, which can effectively improve the encryption effect and avoid the risk of data leakage due to key loss.

[0066] In the embodiment of the present disclosure, the specific method steps for encrypting a compressed file using a digital envelope algorithm and a block encryption algorithm are as follows: The first step is to use the session key to encrypt the compressed file to obtain the ciphertext.

[0067] The session key can be used to encrypt the compressed file using a symmetric encryption algorithm or an asymmetric encryption algorithm.

[0068] The second step is to use a shuffle algorithm to generate a replacement table, which includes multiple bytes.

[0069] The number of bytes in the substitution table may be 256. By generating the substitution table through a shuffling algorithm, the randomness of the substitution table can be ensured, so as to ensure that pattern recognition and substitution are unpredictable.

[0070] Step 3: Divide the ciphertext into multiple blocks in a first preset order according to the preset block size; when the ciphertext data in the last block is smaller than the block size, fill the last block with padding bytes.

[0071] Exemplarily, the preset block size may be 4 bytes (32 bits). The preset block size may affect the efficiency and security of encryption. Generally, the preset block size may also be 8, 16 or 32 bits.

[0072] Exemplarily, the padding bytes may be 00x0.

[0073] Step 4: Replace the bytes in each block with the bytes in the replacement table, and reverse the order of the bytes in the block after the replacement is completed.

[0074] Step 5: Connect the multiple blocks according to the second preset order to obtain a first encrypted file.

[0075] Step 6: Encrypt the session key according to the public key of the recipient to obtain a second encrypted file; the first encrypted file, the second encrypted file and the replacement table constitute an encrypted file.

[0076] In the disclosed embodiment, the ciphertext is further encrypted by the block encryption algorithm, which can ensure that when the session key is leaked, other irrelevant persons cannot decrypt the encrypted file by using the leaked session key to obtain the file to be encrypted. Alternatively, even if the encrypted file is decrypted by using the session key, the information recorded in the decrypted file is different from that in the file to be encrypted.

[0077] S6. Transmit the encrypted file and record table to the recipient through the message middleware.

[0078] In the embodiment of the present disclosure, step S6 includes: The first step is to generate a message queue and set the persistence of the queue to 1.

[0079] The second step is to upload the encrypted file to the message queue, which will send the encrypted file to the recipient.

[0080] In the third step, the receiver sends a receipt message to the sender through the message queue based on the record table and received from the encrypted file.

[0081] In the disclosed embodiment, the name, message type, message header, message information attributes and other parameters of the good message queue are defined, and the persistence of the message queue is set to 1, which can ensure that the transmitted data can be restored even if the system crashes or loses power. Then the priority of the message queue is set, and some urgent files can be sent out first, and finally these contents are placed in the message queue. When a file is sent to the message queue, the message queue will send the file to the recipient, and the entire sending and receiving process is asynchronous, thereby improving the efficiency and throughput of file sending.

[0082] In the disclosed embodiment, when performing data reading, data compression, data encryption and data transmission, useless files are deleted every time a process is completed. For example, when the data compression process is completed, the compressed file has been encrypted into an encrypted file, and the compressed file is useless at this time, so the compressed file is deleted.

[0083] In the disclosed embodiment, after the recipient receives the encrypted file, it will decrypt it by implementing the agreed key (that is, the recipient's private key), thereby ensuring the security of the file information. After the decryption is completed, the recipient will proofread the file to ensure the reliability of the file, and then notify the sender of the received file information. This process is also sent to the sender through a message queue and this process is also encrypted. After the sender receives the receipt information, it will send a message to the recipient again to indicate that the receipt information has been received.

[0084] In the disclosed embodiment, after the recipient receives the encrypted file, the decryption process of the encrypted file is as follows: The receiver decrypts the second encrypted file according to the receiver's private key to obtain a session key; According to the preset block size, the first encrypted file is divided according to the second preset order to obtain a plurality of decrypted blocks; Replace the bytes in the decrypted block according to the inverse of the substitution table and reverse the byte order after the replacement is completed; Combining the multiple decryption blocks according to a first preset order to obtain a first decrypted file; removing padding characters from the first decrypted file; Decrypting the first decryption file using the ciphertext to obtain a compressed file; Decompress the compressed file to obtain the file to be compressed; The data in the file to be compressed is rearranged according to the record table to obtain the data to be transmitted.

[0085] That is, the receiver decrypts the encrypted file according to the rules agreed upon in advance with the sender, as well as the session key and substitution table sent by the sender, thereby decrypting the unencrypted information. At the same time, the receiver encrypts the information using the pre-agreed rules, the digital envelope algorithm and the block encryption algorithm to ensure security.

[0086] After receiving the receipt information, the sender confirms whether the sent file is complete through the status code in the receipt information. For files that fail to send, the file information is stored in the previous data status table, and the sending status is set to failure. When the failed data is scanned again, the file sending process is repeated.

[0087] In the disclosed embodiment, the entire process of data reading, data compression, data encryption and data transmission can be directly recorded in the overview data table and the status table, and displayed to the user through the Web interface.

[0088] In the disclosed embodiment, the user can easily select the data to be transmitted or the specific file in the data table to be sent through the Web interface. The interface is friendly and convenient for users to quickly locate and select the required content.

[0089] In the disclosed embodiment, once the file to be transferred is selected and the transfer begins, the system will automatically record all relevant information of the data file transfer (such as file name, size, transfer time, target address, etc.) into a dedicated database table. This step ensures the traceability of the transfer process.

[0090] In the disclosed embodiment, the user can track the current status of the file transfer in real time through the web page at any time, including the transfer progress, whether the target location is successfully reached, and the recipient's receipt information, etc. This function provides great convenience and transparency.

[0091] In the disclosed embodiment, during the file transfer process, if any failure or transmission failure occurs, the system will immediately record a detailed error log and prompt the user. The user can view these logs to analyze the specific reasons for the failure, such as network problems, insufficient file permissions, or target server failure.

[0092] In the disclosed embodiment, after analyzing the cause of the failure and solving the related problems, the user only needs to click the "Resend" button, and the system will try to transfer the previously failed file again. This function greatly improves the reliability and efficiency of file transfer.

[0093] The present invention first obtains the total number of data items of a certain size through pre-processing of the data, makes a threshold according to the total number of data items, then connects the processes of data reading, data compression, data encryption, and data transmission, and finally processes all the fragments in parallel using the above processes, thereby reducing the waiting time caused by an error in an intermediate step in the overall transmission process and making the best use of the network bandwidth, reducing the probability of packet loss in the large file transmission process, reducing memory loss, being able to more easily locate problems in the transmission process and realizing breakpoint resumption according to the starting position of the fragment, effectively improving the overall data transmission efficiency; the data encryption process is first encrypted by a digital envelope, and then a block replacement algorithm is used to transform and reverse the bytes of a single data fragment, effectively avoiding the risk of data leakage due to key leakage.

[0094] Figure 2 This is a structural block diagram of a secure and efficient data transmission device provided by an embodiment of the present disclosure. Figure 2 ,include: A generation module 11 is used to generate a database, which includes one or more data tables, and the data tables are used to store data to be transmitted; define a thread pool, which includes multiple threads; generate a blank file and a record table, each thread reads the data to be transmitted from the database according to the size of the blank file, and writes the data to be transmitted into the blank file to obtain a file to be compressed; the record table is used to record the name of the file to be compressed, the size of the file to be compressed, the name of the data table and the position of the data to be transmitted in the table; The file processing module 12 is used to compress the file to be compressed to obtain a compressed file; and encrypt the compressed file to obtain an encrypted file; The transmission module 13 is used to transmit the encrypted file and the record table to the receiving party.

[0095] Figure 3 This is a structural block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 3 , electronic devices may include Figure 2 The device for constructing the tunnel disaster warning model. Generally, the electronic device includes: a processor 21 and a memory 22.

[0096] The processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor, the main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. The memory 22 may include one or more computer-readable storage media, which may be non-transitory. The memory 22 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 22 is used to store at least one instruction, which is used to be executed by the processor 21 to implement the method for constructing a tunnel disaster warning model performed by an electronic device provided in the method embodiment of the present application.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A safe and efficient data transmission method, characterized in that: include: Generate a database, the database includes one or more data tables, and the data tables are used to store the data to be transmitted; Define a thread pool, which includes multiple threads; Generate a blank file and a record table. Each thread reads the data to be transmitted from the database according to the size of the blank file, and writes the data to be transmitted into the blank file to obtain the file to be compressed. The record table is used to record the name of the file to be compressed, the size of the file to be compressed, the name of the data table and the position of the data to be transmitted in the table; Compress the file to be compressed to obtain a compressed file; Encrypt the compressed file to obtain an encrypted file; Transmit the encrypted file and log sheet to the recipient.

2. The safe and efficient data transmission method according to claim 1, characterized in that: Each thread reads data from the database according to the size of the blank file and writes the data into the blank file to obtain the file to be compressed, including: Determine the standard size of a single piece of data based on the type of data stored in the data table; Determine the standard number of data items that a blank file can store based on the standard size; Randomly extract multiple data to be transmitted from the data table according to the standard number of data; Based on the total size of the extracted multiple pieces of data to be transmitted, determining the deviation between the total size of the multiple pieces of files to be transmitted and the size of the blank file; Adjust the number of standard data items according to the deviation; Read data from the data table according to the standard number of data entries and write them into a blank file.

3. The safe and efficient data transmission method according to claim 1, characterized in that: Read data from the data table according to the standard number of data entries and write them into a blank file, including: Using cursor processing, data is read from the data table according to the standard number of data entries and written into a blank file.

4. The safe and efficient data transmission method according to claim 1, characterized in that: Encrypt the compressed file to obtain the encrypted file, including: The digital envelope algorithm and the block encryption algorithm are used to encrypt the compressed file to obtain an encrypted file.

5. The safe and efficient data transmission method according to claim 4, characterized in that: The compressed file is encrypted using the digital envelope algorithm and the block encryption algorithm to obtain an encrypted file, including: The compressed file is encrypted using the session key to obtain the ciphertext; A replacement table is generated by using a shuffling algorithm, wherein the replacement table includes a plurality of bytes; According to a preset block size, the ciphertext is divided into a plurality of blocks in a first preset order; when the ciphertext data in the last block is smaller than the block size, the last block is padded with padding bytes; Replace the bytes in each block with the bytes in the replacement table and reverse the order of the bytes in the block after the replacement is completed; Connecting the multiple blocks according to a second preset order to obtain a first encrypted file; The session key is encrypted according to the public key of the receiver to obtain a second encrypted file; the first encrypted file, the second encrypted file and the replacement table constitute an encrypted file.

6. The safe and efficient data transmission method according to claim 5, characterized in that: Transmit the encrypted file and record table to the recipient through the message middleware, including: Generate a message queue and set the queue's persistence to 1; Upload the encrypted file to the message queue, and the message queue sends the encrypted file to the recipient; The receiver sends a receipt message to the sender through the message queue based on the encrypted file and record table.

7. The safe and efficient data transmission method according to claim 6, characterized in that: After the receiver receives the encrypted file, the decryption process of the encrypted file is as follows: The receiver decrypts the second encrypted file according to the receiver's private key to obtain a session key; According to the preset block size, the first encrypted file is divided according to the second preset order to obtain a plurality of decrypted blocks; Replace the bytes in the decrypted block according to the inverse of the substitution table and reverse the byte order after the replacement is completed; Combining the multiple decryption blocks according to a first preset order to obtain a first decrypted file; removing padding characters from the first decrypted file; Decrypting the first decryption file using the ciphertext to obtain a compressed file; Decompress the compressed file to obtain the file to be compressed; The data in the file to be compressed is rearranged according to the record table to obtain the data to be transmitted.

8. A safe and efficient data transmission device, characterized in that: include: A generating module, used for generating a database, wherein the database includes one or more data tables, and the data tables are used for storing data to be transmitted; Define a thread pool, which includes multiple threads; Generate a blank file and a record table. Each thread reads the data to be transmitted from the database according to the size of the blank file, and writes the data to be transmitted into the blank file to obtain the file to be compressed. The record table is used to record the name of the file to be compressed, the size of the file to be compressed, the name of the data table and the position of the data to be transmitted in the table; A file processing module, used for compressing the file to be compressed to obtain a compressed file; Encrypt the compressed file to obtain an encrypted file; The transmission module is used to transmit the encrypted file and the record table to the receiver.

9. An electronic device, characterized in that: The electronic device comprises the safe and efficient data transmission device as described in claim 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program code, and the program code is executed by a processor to implement the secure and efficient data transmission method as described in any one of claims 1 to 7.

Images