Data lossless compression and encryption transmission method based on joint middleware

Abstract

This invention discloses a lossless data compression and encrypted transmission method based on federated middleware, belonging to the field of computer application technology. The method includes: performing run-length encoding on the data to be processed based on federated middleware to obtain first compressed data; processing the first compressed data using a compression algorithm based on a raw-middleware dual dictionary to obtain second compressed data. This compression algorithm uses both the raw dictionary and the middleware dual dictionary, adding only values ​​that appear at a set frequency to the dictionary; sequentially performing Huffman coding and DES processing on the second compressed data to obtain encrypted data and a key; performing RSA processing on the key to obtain an encrypted key; and transmitting the encrypted data and the encrypted key to the target end using an encrypted compression transmission plugin of an experiment-and-train-enabled architecture. This method achieves lossless data compression and encryption based on federated middleware, with fast algorithm speed and high security.

Description

Technical Field

[0001] This invention belongs to the field of computer application technology, and more specifically, relates to a method for lossless data compression and encrypted transmission based on federated middleware. Background Technology

[0002] Traditional data encryption technologies have matured. However, the amount of data transmitted by co-simulation middleware in the experiment and training enabled architecture mode is extremely large, far exceeding the maximum threshold of existing dictionary index compression algorithms. Traditional dictionary compression algorithms will become full due to the exponential growth of the dictionary index, making it impossible to add new data to the dictionary and causing the compression algorithm to fail.

[0003] During encryption, the complexity of the encryption algorithm increases with the amount of data. The failure of the compression algorithm can lead to a decrease in the performance of subsequent encryption algorithms, causing the performance of asymmetric encryption algorithms in subsequent middleware transmission to fall below the minimum encryption performance specifications. Therefore, it is of great significance to address the issue of existing compression algorithms failing when applied to middleware transmission with extremely large data volumes. Summary of the Invention

[0004] In response to the shortcomings and improvement needs of existing technologies, this invention provides a lossless data compression and encrypted transmission method based on co-simulation middleware, which aims to solve the problems of inapplicability and incoordination of data encryption algorithms and data compression algorithms in co-simulation middleware.

[0005] To achieve the above objectives, according to one aspect of the present invention, a method for lossless data compression and encrypted transmission based on federated middleware is provided, comprising: S1, performing run-length encoding on the data to be processed based on federated middleware to obtain first compressed data; S2, processing the first compressed data using a compression algorithm based on a raw-middleware dual dictionary to obtain second compressed data, wherein the compression algorithm based on a raw-middleware dual dictionary uses two dictionaries, a raw dictionary and a middleware dual dictionary, and only adds values ​​that appear at a set frequency to the dictionary; S3, sequentially performing Huffman coding and DES processing on the second compressed data to obtain encrypted data and a key, and performing RSA processing on the key to obtain an encrypted key; S4, transmitting the encrypted data and the encrypted key to a target end using an encrypted compression transmission plugin of an experiment-and-train-enabled architecture.

[0006] Furthermore, S2 includes: S21, creating an initial original dictionary, middleware dictionary, first prefix, and second prefix, and filling the original dictionary with the first compressed data; S22, updating the first prefix to a string obtained by combining the current first prefix and the i-th character of the first compressed data, where i is initially 1; S23, if the updated first prefix does not exist in the middleware dictionary, and the index of the updated first prefix is ​​mistakenly added to the middleware dictionary, adding a key-value pair consisting of the last character of the updated first prefix and its index to the middleware dictionary, and adding a key-value pair consisting of the other characters of the updated first prefix besides the last character and their indices to the middleware dictionary; otherwise, adding a key-value pair consisting of the updated first prefix and its index to the middleware dictionary; S24, combining the second prefix and the first compressed data... The i-th character combination yields the second string. If the second string exists in the original dictionary, the second prefix is ​​updated to the second string, the index of the second prefix is ​​updated to i, and the occurrence count of the second string in the original dictionary is incremented by 1. When its occurrence count reaches the set frequency, the second prefix before the update and its index are combined into a key-value pair and added to the original dictionary. Otherwise, the second prefix is ​​updated to the i-th character of the first compressed data, the second string and its index are combined into a key-value pair and added to the original dictionary, and its occurrence count is set to 1. S25, i is incremented by one, and S22-S24 are repeated until i reaches its maximum value. S26, the newly obtained original dictionary and middleware dictionary are merged to obtain a new dictionary. The first compressed data is compressed based on the new dictionary to obtain the second compressed data.

[0007] Furthermore, S21 includes: creating an empty original dictionary and an empty middleware dictionary, and creating two empty strings as a first prefix and a second prefix respectively, and using a string table compression algorithm to fill the first compressed data into the original dictionary.

[0008] Furthermore, in S23, when the difference between the index of the i-th character of the first compressed data and the length of the updated first prefix is ​​not greater than i, the index of the updated first prefix is ​​mistakenly added to the middleware dictionary.

[0009] Furthermore, before step S4, the process includes: initializing the encrypted compression transmission plugin, and sequentially allocating memory, setting transmission plugin parameters, and setting transmission plugin interface listeners for the encrypted compression transmission plugin; obtaining a list of available interfaces for the transmission plugin instance, and passing the function pointers of each instance of the encrypted compression transmission plugin to the instance's parent class; when the number of network interfaces started is greater than 0, obtaining the last four bits of the interface address to set the ID number of the encrypted compression transmission plugin, and returning the encrypted compression transmission plugin.

[0010] According to another aspect of the present invention, a lossless data compression and encrypted transmission system based on federated middleware is provided, comprising: a first compression module for performing run-length encoding on data to be processed based on federated middleware to obtain first compressed data; a second compression module for processing the first compressed data using a compression algorithm based on a raw-middleware dual dictionary to obtain second compressed data, wherein the compression algorithm based on a raw-middleware dual dictionary uses two dictionaries, a raw dictionary and a middleware dual dictionary, and only adds values ​​that appear at a set frequency to the dictionary; a compression encryption module for sequentially performing Huffman coding and DES processing on the second compressed data to obtain encrypted data and a key, and performing RSA processing on the key to obtain an encrypted key; and a transmission module for transmitting the encrypted data and the encrypted key to a target end using an encrypted compression transmission plug-in of an experiment-and-train-enabled architecture.

[0011] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory storing a computer-executable program, which, when executed by the processor, causes the processor to perform the lossless data compression and encrypted transmission method based on federated middleware as described above.

[0012] According to another aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the lossless data compression and encrypted transmission method based on federated middleware as described above.

[0013] In summary, the above-described technical solutions of this invention achieve the following beneficial effects: A compression algorithm based on a dual dictionary (original and middleware) is designed, where a value is added to the dictionary only when its occurrence frequency reaches a set value, and a structured data structure is used as the dictionary, thus improving the compression ratio; Combining the designed compression algorithm based on the dual dictionary with other lossless compression algorithms enables lossless compression of extremely large object model data, improving the compression ratio and solving the problem of compression algorithm failure when applied to middleware transmission with extremely large data volumes; A compression transmission plugin based on a joint middleware is designed, reducing the amount of data transmitted over the network by the application and improving bandwidth utilization. Attached Figure Description

[0014] Figure 1 A flowchart of a lossless data compression and encrypted transmission method based on federated middleware provided in an embodiment of the present invention;

[0015] Figure 2 A flowchart of a compression algorithm based on a raw-middleware dual dictionary provided in an embodiment of the present invention;

[0016] Figure 3 A block diagram of a lossless data compression and encrypted transmission system based on federated middleware provided in an embodiment of the present invention;

[0017] Figure 4 A block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0019] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0020] Figure 1 A flowchart illustrating a lossless data compression and encrypted transmission method based on federated middleware, provided in an embodiment of the present invention. (See also...) Figure 1 , combined Figure 2 The lossless data compression and encrypted transmission method based on the joint middleware in this embodiment is described in detail. The method includes operations S1-S4.

[0021] Operation S1 performs run-length encoding on the data to be processed based on the federated middleware to obtain the first compressed data.

[0022] Run-length encoding is a statistical encoding method that primarily detects repeating sequences of bits or characters and replaces them with their frequency of occurrence.

[0023] Operation S2 uses a compression algorithm based on a raw-middleware dual dictionary to process the first compressed data to obtain the second compressed data. The compression algorithm based on a raw-middleware dual dictionary uses two dictionaries, the raw dictionary and the middleware dual dictionary, and only adds values ​​that appear at a set frequency to the dictionary.

[0024] According to an embodiment of the present invention, operation S2 includes sub-operations S21-S26, such as... Figure 2 As shown.

[0025] In sub-operation S21, the initial raw dictionary, middleware dictionary, first prefix and second prefix are created, and the first compressed data is filled into the raw dictionary.

[0026] According to an embodiment of the present invention, sub-operation S21 specifically includes: creating an empty original dictionary and an empty middleware dictionary, creating two empty strings as a first prefix and a second prefix respectively, and filling the original dictionary with the first compressed data using a string table compression algorithm.

[0027] In suboperation S22, the first prefix is ​​updated to the string obtained by combining the current first prefix and the i-th character of the first compressed data, where i is initially 1.

[0028] In sub-operation S23, if the updated first prefix does not exist in the middleware dictionary, and the index of the updated first prefix is ​​mistakenly added to the middleware dictionary, the last character of the updated first prefix and its index are combined into a key-value pair and added to the middleware dictionary, and the other characters of the updated first prefix and their indices are combined into a key-value pair and added to the middleware dictionary; otherwise, the updated first prefix and its index are combined into a key-value pair and added to the middleware dictionary.

[0029] According to an embodiment of the present invention, the index of the updated first prefix is ​​determined to be incorrectly added to the middleware dictionary based on the following criteria: when the difference between the index of the i-th character of the first compressed data and the length of the updated first prefix is ​​not greater than i, the index of the updated first prefix is ​​incorrectly added to the middleware dictionary; otherwise, the index of the updated first prefix is ​​not incorrectly added to the middleware dictionary.

[0030] In sub-operation S24, the second prefix and the i-th character of the first compressed data are combined to obtain the second string. If the second string exists in the original dictionary, the second prefix is ​​updated to the second string, the index of the second prefix is ​​updated to i, the occurrence count of the second string in the original dictionary is incremented by 1, and when its occurrence count reaches a set frequency, the second prefix before the update and its index are combined into a key-value pair and added to the original dictionary. Otherwise, the second prefix is ​​updated to the i-th character of the first compressed data, the second string and its index are combined into a key-value pair and added to the original dictionary, and its occurrence count is set to 1.

[0031] In sub-operation S25, i is incremented by one, and sub-operations S22-S24 are executed repeatedly until i reaches its maximum value, thus completing the mapping of the last character of the first compressed data to the dictionary.

[0032] In sub-operation S26, the newly obtained original dictionary and middleware dictionary are merged to obtain a new dictionary. The first compressed data is then compressed based on the new dictionary to obtain the second compressed data.

[0033] Operation S3 sequentially performs Huffman coding and DES processing on the second compressed data to obtain encrypted data and a key. The key is then processed by RSA to obtain the encrypted key.

[0034] Specifically, the second compressed data is Huffman encoded to obtain the third compressed data; the third compressed data is then processed using the Data Encryption Standard (DES) to obtain encrypted data D. t and key K d ; for key K d Perform RSA encryption to obtain the encrypted key K. r .

[0035] According to an embodiment of the present invention, before performing operation S4, it is also necessary to initialize and configure the encrypted compression transmission plugin, including: initializing the encrypted compression transmission plugin, and sequentially allocating memory for the encrypted compression transmission plugin, setting transmission plugin parameters, and setting transmission plugin interface listening; obtaining a list of available interfaces of the transmission plugin instance, and passing the function pointers of each instance of the encrypted compression transmission plugin instance to the parent class of the instance; when the network interface started is greater than 0, obtaining the last four bits of the interface address to set the ID number of the encrypted compression transmission plugin, and returning the encrypted compression transmission plugin.

[0036] The specific execution process of initializing and configuring the encrypted compression transmission plugin is as follows: Initialize the encrypted compression transmission plugin and allocate memory for the transmission plugin instance me; if instance me is null, return NULL, indicating that the creation of the encrypted compression transmission plugin failed and the process ends; if instance me is not null, set the encrypted transmission plugin parameters; if the transmission plugin parameters are not valid, return NULL, indicating that the creation of the encrypted compression transmission plugin failed and the process ends; if the transmission plugin parameters are valid, set the transmission plugin interface listener; if the transmission plugin interface listener is not valid, return NULL, indicating that the creation of the encrypted compression transmission plugin failed and the process ends; if the transmission plugin interface listener is valid, determine whether the list of available interfaces for the transmission plugin instance has been obtained and whether initialization was successful; if the list of available interfaces for the transmission plugin instance has not been obtained or initialization failed, return NULL, indicating that the creation of the encrypted compression transmission plugin failed and the process ends; if the list of available interfaces for the transmission plugin instance has been obtained and initialization was successful, pass the function pointers of the encrypted compression transmission plugin instance to the parent class of the instance; determine whether the started network interface is greater than 0; if not, do not do anything; if greater, obtain the last four digits of the interface address, set the encrypted compression transmission plugin ID number, and return the transmission plugin instance me.

[0037] Operation S4 utilizes the encrypted compression transmission plugin of the experiment-training enabled architecture to transmit encrypted data D. t and the encrypted key K r Transmitted to the target end.

[0038] Figure 3This is a block diagram of a lossless data compression and encrypted transmission system based on federated middleware, provided in an embodiment of the present invention. (See also...) Figure 3 The lossless data compression and encryption transmission system 300 based on the joint middleware includes a first compression module 310, a second compression module 320, a compression encryption module 330, and a transmission module 340.

[0039] The first compression module 310, for example, performs operation S1 to perform run-length encoding on the data to be processed based on the federated middleware to obtain the first compressed data.

[0040] The second compression module 320, for example, performs operation S2 to process the first compressed data using a compression algorithm based on a raw-middleware dual dictionary to obtain the second compressed data. The compression algorithm based on a raw-middleware dual dictionary uses two dictionaries, a raw dictionary and a middleware dual dictionary, and only adds values ​​that appear at a set frequency to the dictionary.

[0041] For example, the compression encryption module 330 performs operation S3 to sequentially perform Huffman encoding and DES processing on the second compressed data to obtain encrypted data and a key, and performs RSA processing on the key to obtain the encrypted key.

[0042] For example, the transmission module 340 performs operation S4 to transmit encrypted data and the encrypted key to the target end using the encrypted compressed transmission plugin of the experiment and training enabled architecture.

[0043] The data lossless compression and encrypted transmission system 300 based on federated middleware is used to perform the above... Figures 1-2 The illustrated embodiment presents a lossless data compression and encrypted transmission method based on a federated middleware. For details not covered in this embodiment, please refer to the foregoing. Figures 1-2 The lossless data compression and encrypted transmission method based on the federated middleware in the illustrated embodiment will not be described in detail here.

[0044] Embodiments of this disclosure also illustrate an electronic device, such as Figure 4 As shown, the electronic device 400 includes a processor 410 and a readable storage medium 420. The electronic device 400 can perform the above-described... Figures 1-2 The method for lossless data compression and encrypted transmission based on federated middleware is described in the paper.

[0045] Specifically, processor 410 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. Processor 410 may also include onboard memory for caching purposes. Processor 410 may be used for executing reference... Figures 1-2The method flow described according to embodiments of this disclosure refers to a single processing unit or multiple processing units performing different actions.

[0046] The readable storage medium 420 can be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, the readable storage medium can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, apparatuses, or propagation media. Specific examples of readable storage media include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); memories such as random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0047] Readable storage medium 420 may include computer program 421, which may include code / computer-executable instructions that, when executed by processor 410, cause processor 410 to perform, for example, the above-described combination. Figures 1-2 The described method and any variations thereof.

[0048] Computer program 421 can be configured to have computer program code, for example, including computer program modules. For example, in an exemplary embodiment, the code in computer program 421 may include one or more program modules, such as 421A, module 421B, ... It should be noted that the division and number of modules are not fixed. Those skilled in the art can use appropriate program modules or combinations of program modules according to the actual situation. When these combinations of program modules are executed by processor 410, processor 410 can perform, for example, the above-described combinations... Figures 1-2 The described method and any variations thereof.

[0049] This invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following... Figures 1-2 The method shown is a lossless data compression and encrypted transmission method based on federated middleware.

[0050] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for lossless data compression and encrypted transmission based on federated middleware, characterized in that, include: S1, perform run-length encoding on the data to be processed based on the joint middleware to obtain the first compressed data; S2, the first compressed data is processed using a compression algorithm based on the original-middleware dual dictionary to obtain the second compressed data. The compression algorithm based on the original-middleware dual dictionary uses two dictionaries, the original dictionary and the middleware dual dictionary, and only adds values ​​that appear at a set frequency to the dictionary. S3, perform Huffman coding and DES processing on the second compressed data in sequence to obtain encrypted data and key, and perform RSA processing on the key to obtain the encrypted key; S4, using the encrypted compressed transmission plugin of the experiment and training enabled architecture, transmit the encrypted data and the encrypted key to the target end; S2 includes: S21, create the initial raw dictionary, middleware dictionary, first prefix and second prefix, and fill the raw dictionary with the first compressed data; S22, update the first prefix to the current first prefix and the first compressed data. A string obtained by combining 10 characters The initial value is 1; S23. If the updated first prefix does not exist in the middleware dictionary, and the index of the updated first prefix is ​​mistakenly added to the middleware dictionary, add a key-value pair consisting of the last character of the updated first prefix and its index to the middleware dictionary, and add a key-value pair consisting of the other characters of the updated first prefix and their indices to the middleware dictionary; otherwise, add a key-value pair consisting of the updated first prefix and its index to the middleware dictionary. S24, combine the second prefix and the first compressed data. The second string is obtained by combining the characters. If the second string exists in the original dictionary, the second prefix is ​​updated to the second string, and the index of the second prefix is ​​updated to... The occurrence count of the second string in the original dictionary is incremented by 1. When its occurrence count reaches the set frequency, a key-value pair consisting of the original second prefix and its index is added to the original dictionary. Otherwise, the second prefix is ​​updated to the first compressed data. The second string and its index are combined into a key-value pair and added to the original dictionary, with their occurrence count set to 1. S25, for Perform an increment process, and repeat steps S22-S24 until... It reaches its maximum value; S26, merge the newly obtained original dictionary and middleware dictionary to obtain a new dictionary, and compress the first compressed data based on the new dictionary to obtain the second compressed data.

2. The data lossless compression and encrypted transmission method based on federated middleware as described in claim 1, characterized in that, S21 includes: creating an empty original dictionary and an empty middleware dictionary, and creating two empty strings as a first prefix and a second prefix respectively, and using a string table compression algorithm to fill the first compressed data into the original dictionary.

3. The lossless data compression and encrypted transmission method based on federated middleware as described in claim 1 or 2, characterized in that, In S23, when the first compressed data... The difference between the index of the character and the length of the updated first prefix is ​​no greater than [value missing]. At that time, the index of the updated first prefix was mistakenly added to the middleware dictionary.

4. The data lossless compression and encrypted transmission method based on federated middleware as described in claim 1, characterized in that, The following is included before S4: Initialize the encrypted compression transmission plugin, and then allocate memory, set transmission plugin parameters, and set transmission plugin interface listening for the encrypted compression transmission plugin in sequence; Get the list of available interfaces for the transmission plugin instance, and pass the function pointers of each encrypted and compressed transmission plugin instance to the parent class of the instance; When the number of network interfaces started is greater than 0, obtain the last four digits of the interface address to set the ID number of the encrypted compression transmission plugin, and return the encrypted compression transmission plugin.

5. A lossless data compression and encrypted transmission system based on federated middleware, characterized in that, include: The first compression module is used to perform run-length encoding on the data to be processed based on the federated middleware to obtain the first compressed data; The second compression module is used to process the first compressed data using a compression algorithm based on the original-middleware dual dictionary to obtain the second compressed data. The compression algorithm based on the original-middleware dual dictionary uses two dictionaries, the original dictionary and the middleware dual dictionary, and only adds values ​​that appear at a set frequency to the dictionary. The compression and encryption module is used to sequentially perform Huffman coding and DES processing on the second compressed data to obtain encrypted data and a key, and to perform RSA processing on the key to obtain an encrypted key. The transmission module is used to transmit the encrypted data and the encrypted key to the target end using the encrypted compression transmission plug-in of the experiment and training enabled architecture; The first compressed data is processed using a compression algorithm based on a raw-middleware dual dictionary to obtain the second compressed data. This algorithm uses both a raw dictionary and a middleware dual dictionary, adding only values ​​that appear at a set frequency to the dictionary. S21, create the initial raw dictionary, middleware dictionary, first prefix and second prefix, and fill the raw dictionary with the first compressed data; S22, update the first prefix to the current first prefix and the first compressed data. A string obtained by combining 10 characters The initial value is 1; S23. If the updated first prefix does not exist in the middleware dictionary, and the index of the updated first prefix is ​​mistakenly added to the middleware dictionary, add a key-value pair consisting of the last character of the updated first prefix and its index to the middleware dictionary, and add a key-value pair consisting of the other characters of the updated first prefix and their indices to the middleware dictionary; otherwise, add a key-value pair consisting of the updated first prefix and its index to the middleware dictionary. S24, combine the second prefix and the first compressed data. The second string is obtained by combining the characters. If the second string exists in the original dictionary, the second prefix is ​​updated to the second string, and the index of the second prefix is ​​updated to... The occurrence count of the second string in the original dictionary is incremented by 1. When its occurrence count reaches the set frequency, a key-value pair consisting of the original second prefix and its index is added to the original dictionary. Otherwise, the second prefix is ​​updated to the first compressed data. The second string and its index are combined into a key-value pair and added to the original dictionary, with their occurrence count set to 1. S25, for Perform an increment process, and repeat steps S22-S24 until... It reaches its maximum value; S26, merge the newly obtained original dictionary and middleware dictionary to obtain a new dictionary, and compress the first compressed data based on the new dictionary to obtain the second compressed data.

6. An electronic device, characterized in that, include: processor; A memory storing a computer-executable program, which, when executed by the processor, causes the processor to perform the lossless data compression and encrypted transmission method based on federated middleware as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the lossless data compression and encrypted transmission method based on the federated middleware as described in any one of claims 1-4.

Images