User data transmission method and system based on distributed network
By converting data into dictionary format and storing and transmitting it in a distributed network, the problem of data transmission efficiency and security of distributed file systems in a high concurrency environment is solved, and efficient data transmission and security guarantee is achieved.
Patent Information
- Application Number
- CN202411164840.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-23
AI Technical Summary
In the environment of high concurrent data request, existing distributed file systems are difficult to improve data transmission efficiency and ensure data security at the same time. They lack optimization and effective compressed file management for different data types, resulting in system performance degradation.
By converting the received data into dictionary format data, including structural attribute features and data content features compressed by formulas, it is stored and restored to the original data format through formulas when downloading, data compression and decompression are realized, and repeated downloads of duplicate parts of data are avoided. , and use dictionary-data storage technology to achieve "slimming" and security guarantees of data.
It improves data transmission efficiency, reduces data access time, enhances data transmission security, avoids system performance degradation, and does not change the transmission mode.
Smart Images

Figure CN119030967A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data transmission, and in particular to a method and system for user data transmission based on a distributed network. Background Art
[0002] In the current information technology field, with the explosive growth of data volume, data storage and access efficiency have become one of the core challenges of distributed system design. Distributed file systems, as key technologies for handling large-scale data storage and access, significantly improve data reliability and access speed by distributing data across multiple nodes. However, in an environment with high concurrent data requests, traditional distributed file systems still face many bottlenecks in response speed and data processing capabilities.
[0003] Data compression technology plays a vital role in distributed systems. By reducing the storage space occupied by data, compression technology can not only reduce storage costs, but also improve the efficiency of data transmission and access to a certain extent. In mainstream operating systems such as Windows, data compression technology has been widely used, but most of them focus on the compression of single files or data blocks, and lack an overall solution optimized for distributed environments.
[0004] Although existing technologies such as Windows data compression schemes perform well in single-node environments, they still have obvious shortcomings in distributed file systems:
[0005] 1. Current technologies mainly focus on data compression itself, that is, reducing the physical size of data through algorithms, but often ignore the targeted optimization of common file types or data features. This "one-size-fits-all" compression method cannot fully utilize the characteristics of different data types, thus limiting the compression efficiency and effect.
[0006] 2. Although existing distributed file systems support data compression, they usually only stay at the level of data compression storage and lack effective management and organization of compressed files. This means that when accessing data, the system needs extra time to decompress the data, which in turn affects the overall access speed.
[0007] 3. As data security becomes increasingly important, data encryption has become an indispensable function. However, existing technologies often find it difficult to achieve effective data encryption while ensuring fast data access. This lack of security makes distributed file systems a potential risk in storing sensitive data.
[0008] 4. In the high-concurrency data request mode, the distributed file system needs to quickly respond to a large number of concurrent requests and provide stable data access services. However, the existing data compression and access mechanisms often cannot meet these requirements at the same time, resulting in a decrease in system performance.
[0009] In view of the above background and technical challenges, it is particularly important to develop a data compression and management solution that can be optimized for the characteristics of distributed file systems. Summary of the invention
[0010] The technical problem to be solved by the present invention is to provide a method and system for user data transmission based on a distributed network, which can improve the data transmission efficiency and ensure the security of data transmission.
[0011] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0012] A method for transmitting user data based on a distributed network, comprising the steps of:
[0013] The server converts the received data from the first client into dictionary format data, wherein the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, and stores the dictionary format data;
[0014] The second client downloads the dictionary format data from the server in response to the download request for the data, and restores the dictionary format data to the original data format through a formula.
[0015] In order to solve the above technical problems, another technical solution adopted by the present invention is:
[0016] A system for transmitting user data based on a distributed network, comprising a first client, a server, and a second client:
[0017] The server converts the received data from the first client into dictionary format data, wherein the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, and stores the dictionary format data;
[0018] The second client downloads the dictionary format data from the server in response to the download request for the data, and restores the dictionary format data to the original data format through a formula.
[0019] The beneficial effect of the present invention is that after receiving the data sent by the first client for the first time, the server converts it into dictionary format data, so as to store the common structural features of the data through the dictionary format data and perform data compression and simplification. When the second client issues a download request for data, the dictionary format data is downloaded from the server, and the dictionary format data is restored to the original data format through a formula. In this way, the second client decompresses the specified data through the dictionary, and there is no need to repeatedly download the repeated part of the data during data transmission; and the data of the same structure have a common structure, and a dictionary-data storage technology is established for it to realize data "slimming" and data dictionary format conversion. At the same time, through data dictionary format conversion instead of data encryption, there is no need to change the transmission mode, which can improve the data transmission efficiency while ensuring the security of data transmission. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 A flowchart of a method for transmitting user data based on a distributed network according to an embodiment of the present invention;
[0021] Figure 2 A schematic diagram of a system for transmitting user data based on a distributed network according to an embodiment of the present invention;
[0022] Figure 3 A flowchart of specific steps of a method for transmitting user data based on a distributed network according to an embodiment of the present invention;
[0023] Figure 4 The present invention is a schematic diagram of dictionary format conversion of a method for transmitting user data based on a distributed network according to an embodiment of the present invention. DETAILED DESCRIPTION
[0024] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.
[0025] Please refer to Figure 1 The embodiment of the present invention provides a method for transmitting user data based on a distributed network, comprising the steps of:
[0026] The server converts the received data from the first client into dictionary format data, wherein the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, and stores the dictionary format data;
[0027] The second client downloads the dictionary format data from the server in response to the download request for the data, and restores the dictionary format data to the original data format through a formula.
[0028] From the above description, it can be seen that the beneficial effect of the present invention is that after the server receives the data sent by the first client for the first time, it converts it into dictionary format data, so as to store the common structural features of the data through the dictionary format data and compress and simplify the data. When the second client issues a download request for the data, the dictionary format data is downloaded from the server, and the dictionary format data is restored to the original data format through a formula. In this way, the second client decompresses the specified data through the dictionary, and there is no need to repeatedly download the repeated part of the data during data transmission; and the data of the same structure have a common structure, and a dictionary-data storage technology is established for it to realize data "slimming" and data dictionary format conversion. At the same time, through data dictionary format conversion instead of data encryption, there is no need to change the transmission mode, which can improve the data transmission efficiency while ensuring the security of data transmission.
[0029] Further, the server converts the received data from the first client into dictionary format data. The server converts the received data from the first client into dictionary format data, and the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, including:
[0030] The server obtains the data of the first client, taking the data format of the data as the structural attribute feature, and taking the data other than the structural attribute feature of the data as the data content feature;
[0031] If the data content feature can be compressed by a formula, formula compressed data is obtained; otherwise, a data storage segment is obtained, and the formula compressed data, the data storage segment and the structural attribute feature are used as dictionary format data.
[0032] From the above description, it can be seen that for data content features that cannot be processed by the dictionary, they are directly generated as data storage segments, and for data content features that can be processed by the dictionary, corresponding formula compressed data are generated. Therefore, the dictionary format data includes both structural attribute features and data storage segments and formula compressed data in the data content features, and can process multiple data types, further improving data transmission efficiency.
[0033] Furthermore, the data format of the data is used as the structural attribute feature, and the data other than the structural attribute feature of the data is used as the data content feature, and then the following is further included:
[0034] The structural attribute characteristics and data content characteristics of the data are stored in corresponding nodes in the form of a binary tree.
[0035] From the above description, it can be seen that the structural attribute features and the data content features are stored in a binary tree to facilitate the subsequent restoration of the dictionary format data.
[0036] Furthermore, storing the dictionary format data includes:
[0037] The formula compressed data is segmented and the segmented formula compressed data is stored in the binary tree nodes.
[0038] From the above description, it can be seen that through segmentation, a dictionary-data-segment storage technology can be established to further achieve data "slimming".
[0039] Furthermore, the dictionary format data is restored to the original data format through a formula, including:
[0040] Generating structural information of corresponding data according to structural attribute characteristics in the dictionary format data;
[0041] The formula compressed data in the dictionary format data is restored, and the restored formula compressed data and the data storage segment are filled into the structural information of the data.
[0042] It can be seen from the above description that generating structural information through structural attribute features in the dictionary format data and filling in the data by restoring the data in the dictionary format data can effectively improve data transmission efficiency.
[0043] Please refer to Figure 2 Another embodiment of the present invention provides a system for transmitting user data based on a distributed network, including a first client, a server, and a second client:
[0044] The server converts the received data from the first client into dictionary format data, wherein the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, and stores the dictionary format data;
[0045] The second client downloads the dictionary format data from the server in response to the download request for the data, and restores the dictionary format data to the original data format through a formula.
[0046] From the above description, it can be seen that the beneficial effect of the present invention is that after the server receives the data sent by the first client for the first time, it converts it into dictionary format data, so as to store the common structural features of the data through the dictionary format data and compress and simplify the data. When the second client issues a download request for the data, the dictionary format data is downloaded from the server, and the dictionary format data is restored to the original data format through a formula. In this way, the second client decompresses the specified data through the dictionary, and there is no need to repeatedly download the repeated part of the data during data transmission; and the data of the same structure have a common structure, and a dictionary-data storage technology is established for it to realize data "slimming" and data dictionary format conversion. At the same time, through data dictionary format conversion instead of data encryption, there is no need to change the transmission mode, which can improve the data transmission efficiency while ensuring the security of data transmission.
[0047] Further, the server converts the received data from the first client into dictionary format data. The server converts the received data from the first client into dictionary format data, and the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, including:
[0048] The server obtains the data of the first client, taking the data format of the data as the structural attribute feature, and taking the data other than the structural attribute feature of the data as the data content feature;
[0049] If the data content feature can be compressed by a formula, formula compressed data is obtained; otherwise, a data storage segment is obtained, and the formula compressed data, the data storage segment and the structural attribute feature are used as dictionary format data.
[0050] From the above description, it can be seen that for data content features that cannot be processed by the dictionary, they are directly generated as data storage segments, and for data content features that can be processed by the dictionary, corresponding formula compressed data are generated. Therefore, the dictionary format data includes both structural attribute features and data storage segments and formula compressed data in the data content features, and can process multiple data types, further improving data transmission efficiency.
[0051] Furthermore, the data format of the data is used as the structural attribute feature, and the data other than the structural attribute feature of the data is used as the data content feature, and then the following is further included:
[0052] The structural attribute characteristics and data content characteristics of the data are stored in corresponding nodes in the form of a binary tree.
[0053] From the above description, it can be seen that the structural attribute features and the data content features are stored in a binary tree to facilitate the subsequent restoration of the dictionary format data.
[0054] Furthermore, storing the dictionary format data includes:
[0055] The formula compressed data is segmented and the segmented formula compressed data is stored in the binary tree nodes.
[0056] From the above description, it can be seen that through segmentation, a dictionary-data-segment storage technology can be established to further achieve data "slimming".
[0057] Furthermore, the dictionary format data is restored to the original data format through a formula, including:
[0058] Generating structural information of corresponding data according to structural attribute characteristics in the dictionary format data;
[0059] The formula compressed data in the dictionary format data is restored, and the restored formula compressed data and the data storage segment are filled into the structural information of the data.
[0060] It can be seen from the above description that generating structural information through structural attribute features in the dictionary format data and filling in the data by restoring the data in the dictionary format data can effectively improve data transmission efficiency.
[0061] The above-mentioned method and system for user data transmission based on a distributed network of the present invention are suitable for improving data transmission efficiency while ensuring the security of data transmission, and are described below through specific implementation methods:
[0062] Embodiment 1
[0063] Please refer to Figure 1 , Figure 3 and Figure 4 , a method for user data transmission based on a distributed network, comprising the steps of:
[0064] S1. The server converts the data received from the first client into dictionary format data, wherein the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, and stores the dictionary format data.
[0065] Specifically, after the first client uploads data to the server for the first time, the server processes the data and converts it into dictionary format data for storage. The server performs the following steps:
[0066] S11. The server obtains the data of the first client, takes the data format of the data as the structural attribute feature, takes the data other than the structural attribute feature as the data content feature, and stores the structural attribute feature and the data content feature of the data in the form of a binary tree in corresponding nodes.
[0067] In this embodiment, after the server obtains the data, it strips the data according to the data structure, that is, obtains the structural attribute characteristics and data content characteristics of the data, and stores both in the form of a binary tree on the corresponding nodes.
[0068] The structural attribute feature is to use the data format as a common feature. For example, the data header of each MP3 file has a common data structure, and the common data structure is used as the structural attribute feature.
[0069] Data content features are the remaining features excluding structural attribute features, and the members in the dictionary are used as feature identifiers for storage.
[0070] S12. If the data content feature can be compressed by a formula, formula compressed data is obtained; otherwise, a data storage segment is obtained, and the formula compressed data, the data storage segment and the structural attribute feature are used as dictionary format data.
[0071] Specifically, the data content features are obtained, and the data formula is matched therefor, and the data content features are simplified according to the data formula. In this embodiment, the formula template has multiple formulas. When matching the data formula, the data can be first loaded into the formula template to obtain the data processed by the formula and compare it with the original data. If the comparison similarity is greater than the threshold, it means that the processed data is still similar to the original data, and the data can be further compressed, that is, the factors in the formula can be adjusted. If the comparison similarity indicates that the data processed by the formula is completely consistent with the original data, it means that the data processed by the formula can be directly simplified, and the corresponding position and formula can be directly generated on the binary tree.
[0072] It also includes:
[0073] The formula compressed data is segmented and the segmented formula compressed data is stored in the binary tree nodes.
[0074] S2. The second client downloads the dictionary format data from the server in response to the download request for the data, and restores the dictionary format data to the original data format through a formula.
[0075] Specifically, according to the structural attribute characteristics in the dictionary format data, the structural information of the corresponding data is generated;
[0076] The formula compressed data in the dictionary format data is restored, and the restored formula compressed data and the data storage segment are filled into the structural information of the data.
[0077] Embodiment 2
[0078] Please refer to Figure 2A system for user data transmission based on a distributed network includes a first client, a server, and a second client. The first client implements each step of a method for user data transmission based on a distributed network in embodiment 1, which is performed by the first client as the execution subject. The server implements each step of a method for user data transmission based on a distributed network in embodiment 1, which is performed by the server as the execution subject. The second client implements each step of a method for user data transmission based on a distributed network in embodiment 1, which is performed by the second client as the execution subject.
[0079] In summary, the present invention provides a method and system for user data transmission based on a distributed network. After receiving the data sent by the first client for the first time, the server converts it into dictionary format data, so as to store the common structural features of the data through the dictionary format data and perform data compression and simplification. When the second client issues a download request for data, the dictionary format data is downloaded from the server, and the dictionary format data is restored to the original data format through a formula. In this way, the second client decompresses the specified data through the dictionary, and there is no need to repeatedly download the repeated part of the data during data transmission; and the data of the same structure have a common structure, and a dictionary-data storage technology is established for it to achieve data "slimming" and data dictionary format conversion. At the same time, through data dictionary format conversion instead of data encryption, there is no need to change the transmission mode, which can improve the data transmission efficiency while ensuring the security of data transmission.
[0080] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A method for transmitting user data based on a distributed network, characterized in that: Includes steps: The server converts the received data from the first client into dictionary format data, wherein the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, and stores the dictionary format data; The second client downloads the dictionary format data from the server in response to the download request for the data, and restores the dictionary format data to the original data format through a formula.
2. A method for transmitting user data based on a distributed network according to claim 1, characterized in that: The server converts the received data from the first client into dictionary format data. The server converts the received data from the first client into dictionary format data, wherein the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, including: The server obtains the data of the first client, taking the data format of the data as the structural attribute feature, and taking the data other than the structural attribute feature of the data as the data content feature; If the data content feature can be compressed by a formula, formula compressed data is obtained; otherwise, a data storage segment is obtained, and the formula compressed data, the data storage segment and the structural attribute feature are used as dictionary format data.
3. A method for transmitting user data based on a distributed network according to claim 2, characterized in that: The data format of the data is used as the structural attribute feature, and the data other than the structural attribute feature of the data is used as the data content feature, and then further includes: The structural attribute characteristics and data content characteristics of the data are stored in corresponding nodes in the form of a binary tree.
4. The method for transmitting user data based on a distributed network according to claim 3, characterized in that: Storing the dictionary format data, including: The formula compressed data is segmented and the segmented formula compressed data is stored in the binary tree nodes.
5. The method for transmitting user data based on a distributed network according to claim 2, characterized in that: The dictionary format data is restored to the original data format through a formula, including: Generating structural information of corresponding data according to structural attribute characteristics in the dictionary format data; The formula compressed data in the dictionary format data is restored, and the restored formula compressed data and the data storage segment are filled into the structural information of the data.
6. A system for user data transmission based on a distributed network, characterized in that: Including the first client, the server and the second client: The server converts the received data from the first client into dictionary format data, wherein the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, and stores the dictionary format data; The second client downloads the dictionary format data from the server in response to the download request for the data, and restores the dictionary format data to the original data format through a formula.
7. A distributed network-based user data transmission system according to claim 6, characterized in that: The server converts the received data from the first client into dictionary format data. The server converts the received data from the first client into dictionary format data, wherein the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, including: The server obtains the data of the first client, taking the data format of the data as the structural attribute feature, and taking the data other than the structural attribute feature of the data as the data content feature; If the data content feature can be compressed by a formula, formula compressed data is obtained; otherwise, a data storage segment is obtained, and the formula compressed data, the data storage segment and the structural attribute feature are used as dictionary format data.
8. A distributed network-based user data transmission system according to claim 7, characterized in that: The data format of the data is used as the structural attribute feature, and the data other than the structural attribute feature of the data is used as the data content feature, and then further includes: The structural attribute characteristics and data content characteristics of the data are stored in corresponding nodes in the form of a binary tree.
9. A distributed network-based user data transmission system according to claim 8, characterized in that: Storing the dictionary format data, including: The formula compressed data is segmented and the segmented formula compressed data is stored in the binary tree nodes.
10. The system for user data transmission based on a distributed network according to claim 7, characterized in that: The dictionary format data is restored to the original data format through a formula, including: Generating structural information of corresponding data according to structural attribute characteristics in the dictionary format data; The formula compressed data in the dictionary format data is restored, and the restored formula compressed data and the data storage segment are filled into the structural information of the data.
Citation Information
Patent Citations
Data compression for communications signalling
CN108293057A
Json data compression method and device based on self-training dictionary coding
CN115941808A
File updating method and device, electronic equipment and storage medium
CN116155919A
Model compression method, terminal deployment method, system and electronic equipment
CN118502709A
System, method for improving efficiency of YANG data model-based network devices
US20230367785A1
