A method for efficient processing and secure transmission of data assets

Abstract

This invention relates to the field of data processing and transmission technology, specifically to a method for efficient processing and secure transmission of data assets. By constructing full JSON data on the server side and transforming it into feature data containing a metadata list and a main data list, and then compressing and encoding it before sending it to the client, the client can decode and decompress the data to restore it. This method significantly improves the efficiency and security of data transmission, solving the problems of high redundancy, slow speed and insufficient security of traditional JSON.

Description

Technical Field

[0001] This invention relates to the field of data processing and transmission technology, specifically to a method for efficient processing and secure transmission of data assets. Background Technology

[0002] JSON (JavaScript Object Notation) is an open standard file and data exchange format. Due to its simplicity and readability, it is widely used in data exchange between the front end and the back end of web applications.

[0003] In actual data exchange, the web application frontend first sends an HTTP / HTTPS request to the server backend's REST interface. Upon receiving the HTTP / HTTPS request, the server backend's REST interface then returns JSON data to the web application frontend. However, this data exchange method has several drawbacks when transmitting large amounts of data, such as:

[0004] ① High data redundancy: JSON format requires each row of data to contain the same column names, even if the corresponding field has no data, the column name still needs to be filled in, resulting in a large amount of duplicate field name information being transmitted, causing data redundancy;

[0005] ② Slow transmission speed: As a text format, JSON experiences significantly increased transmission time when dealing with large amounts of data. This performance gap is particularly pronounced in environments with limited network bandwidth or weak computing power, such as mobile devices or low-power devices, severely impacting data transmission efficiency.

[0006] ③ Insufficient security: JSON data is usually transmitted in plaintext, making it easy for the data to be stolen or tampered with, resulting in a high risk of data asset leakage and making it difficult to meet the data security protection requirements.

[0007] Therefore, in view of the above-mentioned shortcomings of the existing JSON data transmission method, there is an urgent need for a data asset processing and transmission method that can effectively reduce data redundancy, improve transmission efficiency and enhance transmission security. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for efficient processing and secure transmission of data assets. This method involves constructing full JSON data on the server side, transforming it into feature data containing a metadata list and a main data list, and then compressing and encoding it before sending it to the client. The client then decodes and decompresses the data to restore it, which significantly improves the efficiency and security of data transmission and solves the problems of high redundancy, slow speed, and insufficient security of traditional JSON.

[0009] The objective of this invention is achieved through the following approach:

[0010] A method for efficiently processing and securely transmitting data assets includes the following steps:

[0011] 1) On the server side, construct the data to be transmitted into full JSON data;

[0012] 2) Modify the structure of the full JSON data to form feature data;

[0013] 3) Compress and encode the feature data to obtain encrypted feature data;

[0014] 4) Transmit encrypted feature data from the server to the client;

[0015] 5) On the client side, the received encrypted feature data is decoded and decompressed to obtain the data to be transmitted.

[0016] Preferably, in step 2), the modification of the structure of the full JSON data to form feature data specifically includes:

[0017] 2-1) Extract all field names from the full JSON data and form a list of field names while preserving the original order, ensuring that no field names are duplicated in the list;

[0018] 2-2) Define the extracted field name list as a metadata list, which serves as the first core component of the feature data and is used to record the name information of all fields;

[0019] 2-3) Iterate through each piece of data in the full JSON data, and extract the field values ​​of the corresponding fields in the order of the field names in the metadata list to form a continuous sequence of field values;

[0020] 2-4) Define the field value sequence as the main data list, as the second core component of the feature data, used to record the corresponding values ​​of all fields in sequence;

[0021] 2-5) Encapsulate the metadata list and the main data list into a JSON object to form feature data.

[0022] Preferably, in step 3), the compression and encoding of the feature data to obtain encrypted feature data specifically includes:

[0023] 3-1) Compress the main data list of the feature data to obtain byte data;

[0024] 3-2) Encode the byte data to obtain a transmittable encrypted string;

[0025] 3-3) Create blank full JSON data and transform it into blank feature data;

[0026] 3-4) Fill the metadata list of the feature data into the metadata list of the blank feature data, fill the encrypted string into the body data list of the blank feature data, and use the blank feature data as the encrypted feature data.

[0027] Preferably, in step 5), the decoding and decompression of the received encrypted feature data specifically includes:

[0028] 5-1) Extract the main data list from the received encrypted feature data;

[0029] 5-2) Decode the main data list to obtain a binary string;

[0030] 5-3) Decode the binary string to obtain the original main data list;

[0031] 5-4) Replace the main data list in the received encrypted feature data with the original main data list obtained by decoding in step 5-3) to obtain the decoded feature data, and use the decoded feature data as the full JSON data sent by the server.

[0032] Preferably, in step 3-1), the compression uses the DEFLATE algorithm, GZIP algorithm, ZLIB algorithm, or Brotli algorithm.

[0033] Preferably, in step 3-2), the encoding uses Base64 encoding, Base32 encoding, or Base16 encoding.

[0034] Preferably, in step 4), the transmission of encrypted feature data from the server to the client is performed using HTTP or RPC.

[0035] Preferably, the specific form of the feature data is: jsonData={meta:[field1,field2,field3...], data:[value1,value2,value3...]}, where jsonData is a variable or data object storing JSON format data, meta is data describing the data structure, field1 is the name of the first field, field2 is the name of the second field, field3 is the name of the third field, data is an ordered array storing field values, value1 is the specific numerical content of the first field name, value2 is the specific numerical content of the second field name, and value3 is the specific numerical content of the third field name.

[0036] The beneficial effects of this invention are as follows:

[0037] A method for efficiently processing and securely transmitting data assets includes the following steps:

[0038] 1) On the server side, construct the data to be transmitted into full JSON data;

[0039] 2) Modify the structure of the full JSON data to form feature data;

[0040] 3) Compress and encode the feature data to obtain encrypted feature data;

[0041] 4) Transmit encrypted feature data from the server to the client;

[0042] 5) On the client side, the received encrypted feature data is decoded and decompressed to obtain the data to be transmitted.

[0043] This invention constructs full JSON data on the server side, transforms it into feature data containing a list of metadata and a list of main data, and then sends it to the client after compression and encoding. The client then decodes and decompresses the data to restore it. The overall process achieves an organic unity of redundancy elimination, efficient transmission, security assurance, and compatibility adaptation, comprehensively solving the problems of high redundancy, slow speed, and insufficient security in traditional JSON transmission. It is the core guarantee for achieving efficient and secure transmission of data assets.

[0044] Preferably, in step 2), the modification of the structure of the full JSON data to form feature data specifically includes:

[0045] 2-1) Extract all field names from the full JSON data and form a list of field names while preserving the original order, ensuring that no field names are duplicated in the list;

[0046] 2-2) Define the extracted field name list as a metadata list, which serves as the first core component of the feature data and is used to record the name information of all fields;

[0047] 2-3) Iterate through each piece of data in the full JSON data, and extract the field values ​​of the corresponding fields in the order of the field names in the metadata list to form a continuous sequence of field values;

[0048] 2-4) Define the field value sequence as the main data list, as the second core component of the feature data, used to record the corresponding values ​​of all fields in sequence;

[0049] 2-5) Encapsulate the metadata list and the main data list into a JSON object to form feature data.

[0050] This invention transforms the entire JSON data structure by extracting unique field names to form a metadata list, aggregating field values ​​sequentially into a main data list, and then encapsulating it into a JSON object. This completely eliminates the redundancy of repeated field name transmission in traditional JSON. Furthermore, the transformed JSON data maintains the separation of field names and values, providing a foundation for subsequent encryption and compression. Preserving the original order ensures that mapping relationships are not disrupted, and the JSON object encapsulation guarantees data integrity and reproducibility. It is also compatible with web frameworks, forming a core foundation for efficient data processing and secure transmission. This compact structure also reduces transmission load, decreases resource consumption in front-end and back-end serialization and parsing, and improves transmission and processing efficiency, making it particularly suitable for vulnerable environments such as mobile devices.

[0051] Preferably, in step 3), the compression and encoding of the feature data to obtain encrypted feature data specifically includes:

[0052] 3-1) Compress the main data list of the feature data to obtain byte data;

[0053] 3-2) Encode the byte data to obtain a transmittable encrypted string;

[0054] 3-3) Create blank full JSON data and transform it into blank feature data;

[0055] 3-4) Fill the metadata list of the feature data into the metadata list of the blank feature data, fill the encrypted string into the body data list of the blank feature data, and use the blank feature data as the encrypted feature data.

[0056] This invention compresses only the main data list and encodes it into a transmittable encrypted string, forming a non-plaintext transmission format. This ensures that the encrypted data still maintains the structured format of "metadata + main data". This format is compatible with existing Web frameworks and REST interface transmission, and can still provide a clear data structure foundation during subsequent decryption. This not only avoids direct leakage of data content and enhances transmission security, but also significantly reduces the amount of data and improves transmission efficiency, achieving an organic unity of efficient transmission and security.

[0057] Preferably, in step 3-1), the compression uses the DEFLATE algorithm, GZIP algorithm, ZLIB algorithm, or Brotli algorithm, which can perform lossless compression of JSON data and completely restore the data. It also has high compression efficiency, which can significantly reduce the amount of data and meet the high-efficiency and secure transmission requirements of this application.

[0058] Preferably, in step 3-2), the encoding adopts Base64 encoding, Base32 encoding, or Base16 encoding to adapt to JSON and REST interface transmission, which can realize non-plaintext transmission to improve security and ensure a balance between transmission efficiency and security.

[0059] Preferably, in step 4), the transmission of encrypted feature data from the server to the client is done via HTTP or RPC, which is compatible with existing Web frameworks, does not require significant modifications to the existing Web application architecture, and is easy to promote and apply.

[0060] Preferably, the specific form of the feature data is: jsonData={meta:[field1,field2,field3...], data:[value1,value2,value3...]}, where jsonData is a variable or data object storing JSON format data, meta is data describing the data structure, field1 is the name of the first field, field2 is the name of the second field, field3 is the name of the third field, data is an ordered array storing field values ​​corresponding to the specific numerical content of the field names in meta, value1 is the specific numerical content of the first field name, value2 is the specific numerical content of the second field name, and value3 is the specific numerical content of the third field name.

[0061] The traditional JSON structure is "jsonData = [{field1:value1,field2:value2...}]", which requires each data entry to repeatedly store the complete field name, resulting in redundancy of Nr×Nf×Lf (Nr is the number of records, Nf is the number of fields, and Lf is the length of the field name). This leads to large data volume, long transmission time, and insufficient security for plaintext transmission. This invention transforms the JSON structure into characteristic data in the form of "jsonData={meta:[field1...], data:[value1...]}". By separating field names from field values, using `meta` to centrally store field names, and using `data` to store the corresponding values ​​for each field name, the redundancy of Nr×Nf×Lf can be directly eliminated. Simultaneously, data processing is focused on the `data` array (the core data carrier). By separately encrypting and compressing `data`, the amount of data transmitted can be reduced, transmission efficiency improved, and it is compatible with low-performance devices.

[0062] The advantages of this invention are as follows:

[0063] ① This invention extracts recurring field name information from the full JSON data and constructs a feature data structure containing a metadata list (meta) and a main data list (data). The metadata list aggregates all field names only once and transmits them, while the main data list aggregates the corresponding field values ​​according to the metadata order. This completely eliminates the redundancy caused by the repeated carrying of field names in each data entry in the traditional JSON format. In practical applications, after the transformation, the data size of 10,000 data assets containing 86 fields decreased from 27.1MB to 10.2MB, a reduction in transmission volume of approximately 62.36%, significantly reducing data redundancy.

[0064] ② This invention, based on eliminating data redundancy, further encrypts and compresses the main data through compression algorithms (such as the DEFLATE algorithm) and encoding (such as Base64 encoding), achieving a data reduction rate of up to 91.51% compared to the original full JSON data transmission. This significant reduction in data volume lowers transmission time, especially in environments with limited network bandwidth or weak computing power, such as mobile devices and low-power devices, effectively improving data transmission efficiency and performance.

[0065] ③ This invention processes the main data into a string format that cannot be directly read by means of compression and encoding, and transmits it in a non-plaintext format. This avoids the problem of data being easily stolen or tampered with due to traditional JSON plaintext transmission, significantly improves the security of data assets during transmission, reduces the risk of data leakage, and meets the needs of data security protection. Attached Figure Description

[0066] Figure 1 This is a flowchart of the present invention;

[0067] Figure 2 This is a schematic diagram of the data transmission process in an embodiment of the present invention;

[0068] Figure 3 This is a schematic diagram illustrating the amount of data in the full JSON data without eliminating redundancy in an embodiment of the present invention;

[0069] Figure 4 This is a schematic diagram illustrating the amount of data after redundancy removal from the full JSON data in an embodiment of the present invention.

[0070] Figure 5 This is a schematic diagram of the data after plaintext data encoding processing in an embodiment of the present invention. Detailed Implementation

[0071] like Figures 1 to 5 As shown, a method for efficient processing and secure transmission of data assets includes the following steps:

[0072] 1) On the server side, construct the data to be transmitted into full JSON data;

[0073] 2) Modify the structure of the full JSON data to form feature data;

[0074] 3) Compress and encode the feature data to obtain encrypted feature data;

[0075] 4) Transmit encrypted feature data from the server to the client;

[0076] 5) On the client side, the received encrypted feature data is decoded and decompressed to obtain the data to be transmitted.

[0077] The following is an example of implementing the above method:

[0078] 1) On the server side, construct the data to be transmitted into full JSON data;

[0079] The existing full data assets are in JSON format as follows:

[0080] The data size of jsonData = [{field1:value1,field2:value2,field3:value3....}]] is calculated using the following formula:

[0081] Sizefull=Nr×(Nf×Lf+Nf×Lv)=Nr×Nf×(Lf+Lv),

[0082] In the formula, Lf is the length of the field name, Nf is the number of fields, Lv is the length of the field value, Nr is the number of data rows, and the space occupied is Sizefull.

[0083] 2) Modify the structure of the full JSON data to form feature data, specifically including:

[0084] 2-1) Extract all field names from the full JSON data to form a unique list of field names, ensuring that the field names are not duplicated and that the original order is preserved;

[0085] 2-2) Define the extracted field name list as a metadata list, which serves as the first core component of the feature data and is used to record the name information of all fields;

[0086] 2-3) Iterate through each piece of data in the full JSON data, and extract the field values ​​of the corresponding fields in the order of the field names in the metadata list (meta) to form a continuous sequence of field values;

[0087] 2-4) Define the aggregated field value sequence as the main data list (data), which serves as the second core component of the feature data and is used to record the corresponding values ​​of all fields in sequence;

[0088] 2-5) Encapsulate the metadata list (meta) and the main data list (data) into a unified JSON object, forming feature data {meta:[field1,field2,field3...], data:[value1,value2,value3...]}.

[0089] The feature data obtained by modifying the JSON data structure is in the form of: jsonData = {meta:[field1,field2,field3...], data:[value1,value2,value3...]},

[0090] The formula for calculating the size of feature data is:

[0091] Sizefull = Sizemeta + Nr × Nf × Lv

[0092] In the formula, Lf is the length of the field name, Nf is the number of fields, Lv is the length of the field value, Nr is the number of data rows, Sizemeta is the list of field names, and Sizefull is the space occupied (i.e., the space occupied by the feature data).

[0093] It can be seen that the modified JSON format reduces the number of Nr×Nf×Lf values ​​compared to the unmodified format.

[0094] It is worth noting that the modification of the JSON data structure in this implementation is mainly to eliminate redundant data.

[0095] like Figure 3 As shown, in actual data transmission, the server sends 10,000 data assets with 86 fields to the client, resulting in a data size of 27.1MB before redundancy elimination. This embodiment modifies the JSON data structure of the 10,000 data assets sent from the server to the client using the aforementioned method, significantly reducing the data size to 10.2MB after redundancy elimination. Figure 4 As shown, the data volume decreased by 27.1MB - 10.2MB = 16.9MB, and the transmission volume reduction rate was approximately 62.36%.

[0096] 3) The feature data is compressed and encoded to obtain encrypted feature data, specifically including:

[0097] 3-1) Compress the main data list of the feature data to obtain byte data, and further reduce the size. The compression uses the DEFLATE algorithm, GZIP algorithm, ZLIB algorithm, or Brotli algorithm, specifically including:

[0098] 3-1-1) In a Java backend environment, first create a ByteArrayOutputStream instance as the underlying storage container for the data;

[0099] 3-1-2) Subsequently, based on the ByteArrayOutputStream byte output stream, the ZipOutputStream library is used to construct the ZipOutputStream compressed output stream, and compressed entries are created through it;

[0100] 3-1-3) When the write() method of ZipOutputStream is called to write data bytes, the stream will automatically use the DEFLATE compression algorithm to encode the data and write the compressed result to the underlying ByteArrayOutputStream.

[0101] 3-1-4) After the data writing is complete, the final compressed byte array ByteDatas can be obtained by calling the toByteArray() method of ByteArrayOutputStream.

[0102] 3-2) Encode the byte data ByteDatas to obtain a transmittable encrypted string, wherein the encoding uses Base64 encoding, Base32 encoding, or Base16 encoding;

[0103] 3-3) Create blank full JSON data and transform it into blank feature data;

[0104] 3-4) Fill the metadata list of the feature data into the metadata list of the blank feature data, fill the encrypted string into the body data list of the blank feature data, and use the blank feature data as the encrypted feature data. The structure of the encrypted feature data (i.e., the new jsonData) is as follows:

[0105] The data structure {meta:[field1,field2,field3...], data:base64String} contains the header information to be transmitted, which encodes the actual transmitted data and improves data security.

[0106] In this embodiment, encoding the plaintext data in the manner described above reduces the size of the 10.2MB data to 2.3MB after encoding. Figure 5 As shown.

[0107] After optimization and plaintext data encoding, the data transmission volume (2.3MB) is reduced by approximately 91.51% compared to the full JSON data (27.1MB), meaning the data transmission volume is significantly reduced, making it particularly suitable for large-scale data transmission in mobile environments.

[0108] 4) Write the encrypted feature data into the requested outputStream and transmit it from the server to the client using HTTP or RPC.

[0109] 5) On the client side, the received encrypted signature data is decoded and decompressed to obtain the data to be transmitted, specifically including:

[0110] 5-1) The client receives the jsonData, obtains the received encrypted feature data {meta:[field1,field2,field3...], data:base64String} according to the format of jsonData.data, and extracts the main data list from it;

[0111] 5-2) Use the `window.atob()` method (with `jsonData.data` as the parameter) to decode the main data list, restore the content of `jsonData.data`, write it into the variable `binaryString`, and then populate it into the `bufferByte` array to obtain the decrypted `bufferByte`. The specific method is as follows:

[0112] var binaryString = window.atob(base64); / / Decodes base64 into a binary string

[0113] var len = binaryString.length;

[0114] var bytes = new Uint8Array(len);

[0115] for (var i = 0; i < len; i++) {

[0116] bytes[i] = binaryString.charCodeAt(i);

[0117] }

[0118] Get bufferByte=bytes.buffer;

[0119] 5-3) For the variable bufferByte obtained above, call the jszip component in Vue to decode it and obtain the original main data list. The specific method is as follows:

[0120] const zip = await JSZip.loadAsync(bufferByte);

[0121] const unzipContent = await zip.file(fileNameInZip).async('string');

[0122] The data obtained from unzipContent using the above method is stored in the allContent variable, which is the original list of main data.

[0123] 5-4) Replace the main data list (i.e. the content of the data variable) in the received encrypted feature data with the original main data list allContent obtained by decoding in step 5-3), and use the received encrypted feature data as the full JSON data sent by the server.

[0124] In summary, this embodiment can be divided into four stages: redundancy elimination, data encryption, data transmission, and data restoration, as detailed below:

[0125] Redundancy elimination stage: Extract redundant information to form feature data.

[0126] Data encryption stage: The main data is encrypted and compressed to form a non-plaintext transmission.

[0127] Data transmission phase: JSON data is returned via REST interface, compatible with web frameworks.

[0128] Data decryption stage: Decrypting the data on the browser side.

[0129] Data restoration phase: Restoring data assets on the browser side.

[0130] Each stage is interconnected and works synergistically. Redundancy is eliminated to significantly reduce data transmission volume, encryption and compression are used to achieve secure transmission of non-plaintext data, and standardized interfaces ensure compatibility with web frameworks. Finally, the data is completely restored on the client side. This effectively solves the problems of high redundancy, slow speed and insufficient security in traditional JSON data transmission. It achieves an organic unity of efficient data asset processing and secure transmission, and is especially suitable for the transmission needs of large amounts of data assets in scenarios with limited network environments such as mobile devices.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications made to the present invention by those skilled in the art without departing from the spirit of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for efficient processing and secure transmission of data assets, characterized in that, Includes the following steps: 1) On the server side, construct the data to be transmitted into full JSON data; 2) Modify the structure of the full JSON data to form feature data; 3) Compress and encode the feature data to obtain encrypted feature data; 4) Transmit encrypted feature data from the server to the client; 5) On the client side, the received encrypted feature data is decoded and decompressed to obtain the data to be transmitted.

2. The method according to claim 1, characterized in that: In step 2), the structural transformation of the full JSON data to form feature data specifically includes: 2-1) Extract all field names from the full JSON data and form a list of field names while preserving the original order, ensuring that no field names are duplicated in the list; 2-2) Define the extracted field name list as a metadata list, which serves as the first core component of the feature data and is used to record the name information of all fields; 2-3) Iterate through each piece of data in the full JSON data, and extract the field values ​​of the corresponding fields in the order of the field names in the metadata list to form a continuous sequence of field values; 2-4) Define the field value sequence as the main data list, as the second core component of the feature data, used to record the corresponding values ​​of all fields in sequence; 2-5) Encapsulate the metadata list and the main data list into a JSON object to form feature data.

3. The method according to claim 1, characterized in that: In step 3), the compression and encoding of the feature data to obtain encrypted feature data specifically includes: 3-1) Compress the main data list of the feature data to obtain byte data; 3-2) Encode the byte data to obtain a transmittable encrypted string; 3-3) Create blank full JSON data and transform it into blank feature data; 3-4) Fill the metadata list of the feature data into the metadata list of the blank feature data, fill the encrypted string into the body data list of the blank feature data, and use the blank feature data as the encrypted feature data.

4. The method according to claim 1, characterized in that: Step 5), specifically, decoding and decompressing the received encrypted feature data, includes: 5-1) Extract the main data list from the received encrypted feature data; 5-2) Decode the main data list to obtain a binary string; 5-3) Decode the binary string to obtain the original main data list; 5-4) Replace the main data list in the received encrypted feature data with the original main data list obtained by decoding in step 5-3) to obtain the decoded feature data, and use the decoded feature data as the full JSON data sent by the server.

5. The method according to claim 3, characterized in that: In step 3-1), the compression uses the DEFLATE algorithm, GZIP algorithm, ZLIB algorithm, or Brotli algorithm.

6. The method according to claim 3, characterized in that: In step 3-2), the encoding uses Base64 encoding, Base32 encoding, or Base16 encoding.

7. The method according to claim 1, characterized in that: In step 4), the encrypted feature data is transmitted from the server to the client using HTTP or RPC.

8. The method according to claim 1, characterized in that: The specific form of the feature data is: jsonData={meta:[field1,field2,field3...], data:[value1,value2,value3...]}, where jsonData is a variable or data object storing JSON format data, meta is data describing the data structure, field1 is the name of the first field, field2 is the name of the second field, field3 is the name of the third field, data is an ordered array storing field values, value1 is the specific numerical content of the first field name, value2 is the specific numerical content of the second field name, and value3 is the specific numerical content of the third field name.

Images