Self-adaptive incremental transmission method of multi-tenant real-time configuration synchronization system

Abstract

The invention provides a self-adaptive incremental transmission method of a multi-tenant real-time configuration synchronization system, which comprises the following steps: S1, after a production end receives a request of a consumption end, reading an incremental configuration data packet, submitting the incremental configuration data packet to a self-adaptive flow splitting module, executing flow splitting behavior based on default avg-packet-size and max-bnt, and returning bn = 1 batch of data packets; s2, the consumption end receives a return message, submits the return message to an adaptive confluence module and then initiates at least bnt network requests according to the total batch number, retry is supported after failure, and during the period, the production end responds to each request and returns a configuration data packet of each batch; according to the invention, through selection and dynamic adjustment of the adaptive compression algorithm, dynamic optimization can be carried out according to the actual characteristics of the data, the network condition and the compression speed, and the real-time performance and the efficiency of data transmission are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly to an adaptive incremental transmission method for a multi-tenant real-time configuration synchronization system. Background Art

[0002] With the continuous development of information technology, the industrial Internet has become an important trend in enterprise informatization construction. The multi-tenant real-time configuration synchronization system is one of the important components for building the travel industrial Internet. The system allows multiple tenants to share the same set of software instances and realizes data independence and security through configuration isolation.

[0003] However, in the multi-tenant real-time configuration synchronization system, due to the large amount of tenant configuration data and frequent updates, traditional data transmission methods are difficult to meet the requirements of real-time and efficiency.

[0004] Although some compression algorithms are used for data transmission in the prior art, they often lack adaptability and cannot dynamically adjust the compression algorithm in multiple dimensions such as data characteristics, network conditions, and compression speed.

[0005] Moreover, in the context of the growing online car-hailing business, configuration changes are becoming more frequent and configuration data packets are becoming more inflated. In addition, the consumer side depends on the full configuration. When a configuration sub-item changes, multi-tenant real-time full configuration synchronization needs to be executed.

[0006] In this context, problems such as large full configuration data packets, low transmission efficiency, and low interface stability have become bottlenecks of the system. An adaptive incremental transmission scheme based on synchronizing real-time configurations of different tenants can reduce the size of the overall data packet, reduce the size of a single data packet, reduce the network transmission time, and can adaptively adjust the split flow granularity based on statistical analysis. The system autonomously balances efficiency and performance to ensure timeliness and stability.

[0007] Normally, the consumer side requests the production side to read the full configuration in a fixed paging manner.

[0008] Problems existing in the existing solutions:

[0009] Each time the full configuration is read, the overall data packet is large, the interface takes a long time, and the bandwidth is occupied.

[0010] The fixed paging method cannot control the size of the network data packet for each request and there is a risk of timeout. Therefore, an adaptive incremental transmission method for a multi-tenant real-time configuration synchronization system is proposed. Summary of the Invention

[0011] In view of this, embodiments of the present invention hope to provide an adaptive incremental transmission method for a multi-tenant real-time configuration synchronization system to solve or alleviate the technical problems existing in the prior art and at least provide a beneficial alternative.

[0012] The technical solution of the embodiment of the present invention is implemented as follows: an adaptive incremental transmission method of a multi-tenant real-time configuration synchronization system comprises the following steps:

[0013] S1. After receiving the request from the consumer, the producer reads the incremental configuration data packet and submits it to the adaptive de-streaming module. It performs de-streaming based on the default avg-packages-size and max-bnt, and returns a batch of data packets with bn=1.

[0014] S2. The consumer receives the returned message, submits it to the adaptive confluence module, and then initiates at least bnt network requests according to the total number of batches. Failure supports retry. During this period, the producer responds to each request and returns each batch of configuration data packets.

[0015] S3. After all batch requests are completed, that is, after all data requests with bn values ​​ranging from [1, bnt] are completed, the consumer-side adaptive confluence module merges all batch configuration data packets, which means that the current sequence (seq) task is completed;

[0016] S4, statistics asynchronously counts the time consumption of multi-tenant interfaces and the size of each batch of configured data packets. Calculate max-bnt and avg-package-size;

[0017] S5. The manager dynamically synchronizes max-bnt and avg-package-size to the production-side adaptive flow splitting module through the push + pull model, so that the system can autonomously adjust the flow splitting granularity.

[0018] S6, data collection: collect configuration data in the multi-tenant system in real time and divide it into multiple data blocks;

[0019] S7, data characteristic analysis: perform characteristic analysis on each data block;

[0020] S8, Adaptive compression selection algorithm: Scores are given based on multiple dimensions such as the characteristic analysis results of the data block, network conditions, and estimated compression speed. A priority list of compression algorithms is generated based on the scores, and then the algorithm with the highest priority is selected for compression;

[0021] S9, data transmission: The compressed data block is transmitted through the network. During the transmission process, the monitoring thread will monitor the network status of the network bandwidth and packet loss rate in real time, and adjust the network status dimension score of the adaptive compression selection algorithm according to the network status, and then dynamically adjust the compression strategy;

[0022] S10, data decompression and synchronization: After receiving the compressed data, the receiving end decompresses it according to the compression algorithm and synchronizes the decompressed data to the local configuration.

[0023] In some embodiments, in S7, the feature analysis of each data block includes data size, data type, and data update frequency.

[0024] In some embodiments, if the detected data is less than the set threshold of 1MB, the source data is directly used without compression.

[0025] In some embodiments, the adaptive compression selection algorithm includes Huffman, LZ77, LZ78, GZIP, DEFLATE, CJSON, or HPack. When compression fails, it is dynamically adjusted to the algorithm with the next highest priority for compression. If all fail, no compression is performed, and the original data is directly used as the compressed data block.

[0026] In some embodiments, when configuring data synchronization, the adaptive compression selection algorithm determines through multi-dimensional weighted scoring that the score of "GZIP" is the highest, so this algorithm is preferentially used for compression.

[0027] In some embodiments, when the monitoring thread detects that the real-time network condition is poor, the score of this dimension of the network condition is increased, thereby dynamically adjusting the adaptive compression strategy and preferentially selecting an algorithm with a high compression ratio and a slow compression speed.

[0028] Due to the above technical solutions, the embodiments of the present invention have the following advantages:

[0029] 1. Through the selection and dynamic adjustment of the adaptive compression algorithm, the present invention can perform dynamic optimization according to the actual characteristics of the data, network conditions, and compression speed, improving the real-time performance and efficiency of data transmission.

[0030] 2. By using multiple compression algorithms for backup, the present invention ensures that when a certain compression algorithm cannot effectively compress, it can switch to other algorithms for compression, thereby improving the stability and reliability of the system.

[0031] 3. After data acquisition, the present invention divides the data into blocks. According to the characteristics of different block data, the most suitable compression algorithm can be respectively used, and parallel compression can be performed, thereby improving the flexibility and efficiency of the system.

[0032] 4. Through this method, the present invention can perform adaptive incremental transmission, and the system autonomously balances efficiency and performance. A new type of general adaptive incremental transmission communication protocol.

[0033] The above summary is only for the purpose of the specification and is not intended to be limiting in any way. In addition to the above-described illustrative aspects, embodiments, and features, further aspects, embodiments, and features of the present invention will be readily apparent by reference to the drawings and the following detailed description. Brief Description of the Drawings

[0034] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0035] Figure 1 is the flowchart of the present invention;

[0036] Figure 2 is the overall flowchart of the adaptive compression transmission of the present invention;

[0037] Figure 3 is the processing flowchart of the adaptive compression selection algorithm of the present invention. Detailed Description of the Embodiments

[0038] In the following, only some exemplary embodiments are briefly described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the spirit or scope of the present invention. Therefore, the drawings and the description are considered to be exemplary in nature rather than restrictive.

[0039] It should be noted that terms such as "first", "second", "symmetric", "array", etc. are only used for the purpose of distinguishing descriptions and position descriptions, and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, features defined with "first", "symmetric", etc. can explicitly or implicitly include one or more of such features; similarly, when certain features are not limited in quantity by words such as "two", "three", etc., it should be noted that such features also belong to explicitly or implicitly including one or more feature quantities;

[0040] In the present invention, unless otherwise clearly specified and defined, terms such as "installation", "connection", "fixation", etc. should be understood in a broad sense; for example, it can be a fixed connection, a detachable connection, or an integral molding; it can be a mechanical connection, a direct connection, a welding connection, or an indirect connection through an intermediate medium, and it can be the internal communication of two components or the interaction relationship between two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the drawings of the specification and the specific circumstances.

[0041] The following will describe the embodiments of the present invention in detail with reference to the drawings.

[0042] As shown in Figures 1 - 3As shown, an embodiment of the present invention provides an adaptive incremental transmission method for a multi-tenant real-time configuration synchronization system, comprising the following steps:

[0043] S1. After receiving the request from the consumer, the producer reads the incremental configuration data packet and submits it to the adaptive de-streaming module. It performs de-streaming based on the default avg-packages-size and max-bnt, and returns a batch of data packets with bn=1.

[0044] S2. The consumer receives the returned message, submits it to the adaptive confluence module, and then initiates at least bnt network requests according to the total number of batches. Failure supports retry. During this period, the producer responds to each request and returns each batch of configuration data packets.

[0045] S3. After all batch requests are completed, that is, after all data requests with bn values ​​ranging from [1, bnt] are completed, the consumer-side adaptive confluence module merges all batch configuration data packets, which means that the current sequence (seq) task is completed;

[0046] S4, statistics asynchronously counts the time consumption of multi-tenant interfaces and the size of each batch of configuration data packets. Calculate max-bnt and avg-package-size;

[0047] S5. The manager dynamically synchronizes max-bnt and avg-package-size to the production-side adaptive flow splitting module through the push + pull model, so that the system can autonomously adjust the flow splitting granularity.

[0048] S6, data collection: collect configuration data in the multi-tenant system in real time and divide it into multiple data blocks;

[0049] S7, data characteristic analysis: perform characteristic analysis on each data block;

[0050] S8, Adaptive compression selection algorithm: Scores are given based on multiple dimensions such as the characteristic analysis results of the data block, network conditions, and estimated compression speed. A priority list of compression algorithms is generated based on the scores, and then the algorithm with the highest priority is selected for compression;

[0051] S9, data transmission: The compressed data block is transmitted through the network. During the transmission process, the monitoring thread will monitor the network status of the network bandwidth and packet loss rate in real time, and adjust the network status dimension score of the adaptive compression selection algorithm according to the network status, and then dynamically adjust the compression strategy;

[0052] S10, data decompression and synchronization: After receiving the compressed data, the receiving end decompresses it according to the compression algorithm and synchronizes the decompressed data to the local configuration.

[0053] In this embodiment, specifically, in S7, the feature analysis of each data block includes data size, data type, and data update frequency.

[0054] In this embodiment, specifically, if it is detected that the data is less than the set threshold of 1MB, the source data is directly used without compression.

[0055] In this embodiment, specifically, the adaptive compression selection algorithm includes Huffman, LZ77, LZ78, GZIP, DEFLATE, CJSON, or HPack. When compression fails, it is dynamically adjusted to the algorithm with the next highest priority for compression. If all fail, no compression is performed, and the original data is directly used as the compressed data block.

[0056] In this embodiment, specifically, when configuring data synchronization, the adaptive compression selection algorithm determines through multi-dimensional weighted scoring that the score of "GZIP" is the highest, so this algorithm is preferentially used for compression.

[0057] In this embodiment, specifically, if the monitoring thread detects that the real-time network condition is poor, the score of this dimension of the network condition is increased, thereby dynamically adjusting the adaptive compression strategy and preferentially selecting an algorithm with a high compression ratio and a slow compression speed.

[0058] In this embodiment, specifically, the adaptive compression selection algorithm is as follows:

[0059]

[0060]

[0061]

[0062]

[0063] In this embodiment, specifically, tenant: A online car-hailing service provider can be regarded as a tenant in the system.

[0064] Adaptive compression: Weighted scoring is performed according to dimensions such as data characteristics, network conditions, and compression speed. The priority of optional compression algorithms is determined according to the scores, and then the compression strategy is dynamically adjusted.

[0065] Compression ratio: Data size before compression / Data size after compression.

[0066] Configuration: Metadata participating in business processes.

[0067] Adaptive: Automatically adapt to the environmental scenario and adopt the optimal strategy.

[0068] Increment: The specific detailed items that actually change in the full configuration.

[0069] Transmission: Based on network data behavior between the consumer side and the production side.

[0070] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various changes or substitutions, which should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An adaptive incremental transmission method for a multi-tenant real-time configuration synchronization system, characterized in that: The following steps are involved: S1. After receiving the request from the consumer, the producer reads the incremental configuration data packet and submits it to the adaptive de-streaming module. It performs de-streaming based on the default avg-packages-size and max-bnt, and returns a batch of data packets with bn=1. S2. The consumer receives the returned message, submits it to the adaptive confluence module, and then initiates at least bnt network requests according to the total number of batches. Failure supports retry. During this period, the producer responds to each request and returns each batch of configuration data packets. S3. After all batch requests are completed, that is, after all data requests with bn values ​​ranging from [1, bnt] are completed, the consumer-side adaptive confluence module merges all batch configuration data packets, which means that the current sequence (seq) task is completed; S4, statistics asynchronously counts the time consumption of multi-tenant interfaces and the size of each batch of configured data packets. Calculate max-bnt and avg-package-size; S5. The manager dynamically synchronizes max-bnt and avg-package-size to the production-side adaptive flow splitting module through the push + pull model, so that the system can autonomously adjust the flow splitting granularity. S6, data collection: collect configuration data in the multi-tenant system in real time and divide it into multiple data blocks; S7, data characteristic analysis: perform characteristic analysis on each data block; S8, Adaptive compression selection algorithm: Scores are given based on multiple dimensions such as the characteristic analysis results of the data block, network conditions, and estimated compression speed. A priority list of compression algorithms is generated based on the scores, and then the algorithm with the highest priority is selected for compression; S9, data transmission: The compressed data block is transmitted through the network. During the transmission process, the monitoring thread will monitor the network status of the network bandwidth and packet loss rate in real time, and adjust the network status dimension score of the adaptive compression selection algorithm according to the network status, and then dynamically adjust the compression strategy; S10, data decompression and synchronization: After receiving the compressed data, the receiving end decompresses it according to the compression algorithm and synchronizes the decompressed data to the local configuration.

2. The adaptive incremental transmission method of the multi-tenant real-time configuration synchronization system according to claim 1 is characterized in that: In S7, each data block is subjected to characteristic analysis including data size, data type and data update frequency.

3. The adaptive incremental transmission method of the multi-tenant real-time configuration synchronization system according to claim 1 is characterized in that: If it is detected that the data is smaller than the set threshold of 1MB, the source data is used directly without compression.

4. The adaptive incremental transmission method of a multi-tenant real-time configuration synchronization system according to claim 1, characterized in that: The adaptive compression selection algorithm includes Huffman, LZ77, LZ78, GZIP, DEFLATE, CJSON or HPack. When compression fails, it is dynamically adjusted to the next priority algorithm compression. When all fail, no compression is performed and the original data is directly used as the compressed data block.

5. The adaptive incremental transmission method of a multi-tenant real-time configuration synchronization system according to claim 1, characterized in that: During configuration data synchronization, the adaptive compression selection algorithm uses multi-dimensional weighted scoring to determine that "GZIP" has the highest score, and this algorithm is used for compression first.

6. The adaptive incremental transmission method of a multi-tenant real-time configuration synchronization system according to claim 1, characterized in that: If the monitoring thread detects that the real-time network condition is poor, the score of the network condition dimension is increased, thereby dynamically adjusting the adaptive compression strategy and giving priority to algorithms with high compression ratios and slow compression speeds.

Images