Large-scale data transmission method and system based on adaptive distributed architecture

Abstract

The invention relates to the technical field of data transmission security, and discloses a large-scale data transmission method and system based on an adaptive distributed architecture, and the method comprises a data preprocessing module which is used for extracting data from a data warehouse and pushing the data to a message theme; the data distribution module is used for classifying the data, and distributing the data according to a specified thread number after segmented encryption compression processing; the transmission optimization module is used for selecting a transmission path and a protocol according to the network parameters; the client feedback adaptive module is used for dynamically adjusting the size of a thread pool and the priority of a thread according to a client receiving state and a server instruction; the data fault-tolerant compensation module is used for checking and compensating data loss; and the scheduler is used for comprehensively analyzing the feedback of the client and the network state, dynamically adjusting the number of specified threads by utilizing a distribution strategy prediction model, dynamically responding to the network change, improving the transmission efficiency, the security and the accuracy, reducing the human intervention demand, and being suitable for a high-concurrency scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data transmission security, and in particular to a large-scale data transmission method and system based on an adaptive distributed architecture. BACKGROUND

[0002] Large-scale data transmission technology is one of the core supporting technologies in the fields of modern distributed computing, cloud computing and Internet of Things. With the explosive growth of enterprise data volume and the global deployment of business, efficient and reliable transmission of data between different nodes has become a key to ensuring system performance. Currently, this technical field mainly relies on distributed system architecture, network protocol optimization and load balancing mechanism to realize data flow.

[0003] In the existing technical system, large-scale data transmission usually adopts a client-server or peer-to-peer network architecture, combined with a traditional TCP / IP protocol stack for data packet transmission. In order to improve transmission efficiency, various technical means are generally adopted in the industry: a load balancer based on fixed rules (such as round robin or minimum connection number algorithm) to distribute network traffic; a static configuration data block strategy (such as fixed size data segmentation) to optimize network utilization; a retransmission mechanism (such as the ARQ protocol of TCP) to ensure data reliability; and a simple path selection algorithm (such as the OSPF or BGP protocol) to determine the transmission route.

[0004] However, these existing technical solutions gradually expose many limitations when dealing with complex and variable actual application scenarios: First, static resource security configuration cannot adapt to dynamically changing network environments. Traditional systems generally use predefined security configurations, such as fixed-period key updates and uniform encryption algorithm suites. This rigid security strategy cannot be adapted to changes in network environments and threat situations. For example, when a man-in-the-middle attack is detected, the system cannot automatically increase the encryption strength or switch the authentication mechanism, resulting in a lag in security protection.

[0005] Second, there is a lack of intelligent transmission decision mechanism. The static parameter configuration of traditional systems, such as fixed thread pool size and predefined data block strategy, lacks adaptability when facing network bandwidth fluctuations and link quality changes. When network congestion occurs, fixed-size data segmentation may exacerbate packet loss; while in good network conditions, available bandwidth cannot be fully utilized, resulting in low transmission efficiency.

[0006] Third, there is a lack of coordination between system components. In the existing architecture, each module operates independently, and there is a lack of effective feedback mechanism between the data distribution module and the transmission optimization module. When the client's receiving capacity changes, this change cannot be fed back to the sending end to adjust the sending strategy in a timely manner, which can easily lead to data accumulation or overload of the receiving end.

[0007] Finally, existing fault tolerance mechanisms suffer from efficiency issues. Traditional end-to-end retransmission results in significant bandwidth waste when dealing with large amounts of data, especially in high-latency networks, where the loss of a single data packet can lead to the retransmission of the entire data block, severely impacting transmission throughput.

[0008] These technical deficiencies are amplified in large-scale data transmission scenarios, especially in cross-regional and cross-network domain data synchronization tasks, manifesting as large fluctuations in transmission latency, unstable throughput, and low resource utilization, which seriously affect the quality and reliability of data services. Summary of the Invention

[0009] To simultaneously improve the efficiency, quality, and security of large-scale data transmission in high-concurrency scenarios, this application provides a method and system for large-scale data transmission based on an adaptive distributed architecture.

[0010] Firstly, this application provides a large-scale data transmission system based on an adaptive distributed architecture, employing the following technical solution: A large-scale data transmission system based on an adaptive distributed architecture includes: The data preprocessing module is used to extract data from the data warehouse and push it to the message topic; The data distribution module is used to classify data, perform segmented encryption and compression processing, and then distribute the data according to the specified number of threads. The transmission optimization module is used to select the transmission path and protocol based on network parameters; The client feedback adaptive module is used to dynamically adjust the thread pool size and thread priority based on the client's received status and the server's instructions. The data fault tolerance and compensation module is used to verify and compensate for data loss; The scheduler is used to comprehensively analyze client feedback and network status, and dynamically adjust the specified number of threads using a distribution strategy prediction model.

[0011] By adopting the above technical solutions and using modular collaborative work, the entire data transmission process is adaptively optimized, which can dynamically respond to network changes and improve transmission efficiency and accuracy. The introduction of the scheduler solves the shortcomings of the static strategy of the traditional system, reduces the need for human intervention, and is suitable for high-concurrency scenarios.

[0012] Optionally, the operation of the data distribution module includes: Obtain the data size label and type label of the data; Configure segment sizes according to the data size labels, and prioritize data distribution according to the type labels; The segment size increases with the data size, and the type labels, in order of priority from low to high, include video stream data, log data, and transaction data.

[0013] Optionally, the segmented encryption and compression process includes: The data is segmented with overlap to obtain n data segments; The relevant data segments are encrypted based on the overlapping data; The encrypted data is then compressed in segments; Specifically, regarding one of the aforementioned data: If the first and non-terminal data segments are encrypted based on the overlapping data at their corresponding ends, then the terminal data segments are encrypted based on the data specified at the beginning of the first data segment. If the first data segment is not encrypted based on the overlapping data of its corresponding front end, then the first data segment is encrypted based on the specified data at the end of the last data segment. The lengths of the specified beginning data and the specified end data are the same as those of the overlapping data.

[0014] Optionally, the data distribution module is also used to monitor the push volume and trigger a flow control mechanism when it exceeds a threshold.

[0015] Optionally, the operation of the transmission optimization module includes: The transmission path and protocol are selected by real-time monitoring of network parameters, including latency, bandwidth, packet loss rate, jitter, and network load. Maintain a list of available paths and calculate a cost score for each path. The cost score is calculated based on the network parameters and their corresponding weights. The lower the score, the better the path. If the cost difference between multiple paths is less than 5%, a load balancing strategy is adopted to distribute the data flow across multiple paths. The transport protocol is dynamically selected based on network parameters and data type labels; protocol options include TCP, UDP, and QUIC. If either the packet loss rate or the jitter parameter exceeds the corresponding threshold, the UDP protocol is selected and forward error correction is enabled to reduce retransmission latency. If the delay is greater than its corresponding threshold and the bandwidth is less than its corresponding threshold, then the QUIC protocol is selected to take advantage of its multiplexing and fast handshake. In other cases, choose the TCP protocol to ensure reliability.

[0016] Optionally, the frequency at which the transmission optimization module adjusts the transmission path and protocol is positively correlated with the variance of the delay over a specified time period in the past.

[0017] Optionally, the client feedback adaptive module collects CPU usage and network bandwidth feedback from the client. When high load is detected, it reduces the number of threads through server-side instructions; at the same time, it uses a thread pool manager to adjust thread priorities to ensure that critical data is processed first.

[0018] Optionally, the data fault tolerance compensation module calculates a checksum after data transmission and compares it with the client's checksum result to quickly identify the location of inconsistent data using the data segmentation; if inconsistent, it triggers the ARQ protocol to retransmit the lost data segment; the number of retransmissions is dynamically adjusted according to network conditions.

[0019] Optionally, the training of the distribution strategy prediction model specifically includes: The data preparation steps include collecting terminal operating environment information and conducting exploratory data analysis and preprocessing; The model building steps include determining explanatory variables and estimating regression coefficients using the least squares method; The model evaluation steps include calculating the coefficient of determination and performing statistical tests. The model optimization steps involve optimizing the model through variable transformation or feature selection. The model application steps involve using the optimized model to make predictions and evaluate its performance. The distribution strategy prediction model caches the predicted number of threads in the Redis database and dynamically adjusts the local concurrent thread count and the terminal ingestion thread count based on the prediction results.

[0020] Secondly, this application provides a large-scale data transmission method based on an adaptive distributed architecture, which runs on the aforementioned large-scale data transmission system based on an adaptive distributed architecture.

[0021] In summary, this application includes at least one of the following beneficial technical effects: First, at the overall system level, this invention achieves a synergistic improvement in transmission efficiency and resource utilization. By introducing an intelligent scheduler and adaptive architecture, the system can dynamically adjust its transmission strategy based on real-time network conditions, data characteristics, and client feedback, completely overcoming the limitations of traditional systems that rely on static configuration. Specifically, this results in reduced transmission latency, increased throughput, and significantly optimized utilization of computing resources and network bandwidth. This overall performance improvement makes the system particularly suitable for handling large-scale data synchronization tasks across regions, significantly reducing operating costs while ensuring timely data transmission.

[0022] Secondly, at the module collaboration level, this invention achieves refined resource scheduling through the intelligent linkage of various functional modules. The data distribution module dynamically adjusts the segmentation strategy based on data type and network conditions, the transmission optimization module intelligently selects paths and protocols based on multi-dimensional parameters, and the client feedback module adjusts thread resources in real time. These modules form an organic whole under the unified coordination of the scheduler. This collaborative mechanism ensures that the system can respond quickly to network fluctuations. For example, when network congestion is detected, the system will coordinate the data distribution module to reduce segment size, the transmission optimization module to switch transmission paths, and the client module to adjust receiving threads within milliseconds, thereby maintaining transmission stability. This close cooperation between modules enables the system to maintain excellent performance even in complex network environments.

[0023] Finally, at the secure transmission level, this invention innovatively achieves a balance between security and transmission efficiency. Through a triple guarantee of dynamic encryption strategies, fine-grained access control, and real-time security monitoring, the system can automatically adjust security configurations in transmission scenarios with different security requirements. For example, strong encryption algorithms are used and key update cycles are shortened for highly sensitive data, while security overhead is appropriately reduced for ordinary data to improve transmission efficiency. Simultaneously, the data fault tolerance compensation module, combined with an intelligent verification mechanism, can reduce retransmission overhead while ensuring data integrity. This refined security design not only meets increasingly stringent data compliance requirements but also avoids the excessive impact of security measures on transmission performance in traditional systems. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the module principle of the large-scale data transmission system based on an adaptive distributed architecture in this application. Detailed Implementation

[0025] The embodiments of this application are described in detail below, and examples of the embodiments are shown in the accompanying drawings.

[0026] In the description of this specification, the references to "certain embodiments," "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples" refer to specific features, structures, materials, or characteristics described in connection with the described embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0027] This application discloses a large-scale data transmission system based on an adaptive distributed architecture. The system is deployed in a cloud computing environment and includes a data preprocessing module, a data distribution module, a transmission optimization module, a client feedback adaptive module, a data fault tolerance compensation module, and a scheduler. These modules work collaboratively through a distributed architecture to achieve adaptive optimization of large-scale data transmission, as described above. Figure 1 ,include: The data preprocessing module extracts data from the Oracle data warehouse using OGG technology, captures data changes, pre-stores customer order data in the cache, and pushes it to the Kafka message topic based on Redis subscription information. Its pre-storage and solidification processes improve the efficiency and reliability of data preparation, reduce the risk of errors in subsequent transmission, and meet the needs of application scenarios with high requirements for data freshness through real-time extraction.

[0028] The data distribution module is used to classify data, perform segmented encryption and compression processing, and then distribute the data according to the specified number of threads. The transmission optimization module is used to select the transmission path and protocol based on network parameters; The client feedback adaptive module is used to dynamically adjust the thread pool size and thread priority based on the client's received status and the server's instructions. The data fault tolerance and compensation module is used to verify and compensate for data loss; The scheduler is used to comprehensively analyze client feedback and network status, and dynamically adjust the specified number of threads using a distribution strategy prediction model.

[0029] By adopting the above technical solutions and using modular collaborative work, the entire data transmission process is adaptively optimized, which can dynamically respond to network changes and improve transmission efficiency and accuracy. The introduction of the scheduler solves the shortcomings of the static strategy of the traditional system, reduces the need for human intervention, and is suitable for high-concurrency scenarios.

[0030] During data transmission, the data distribution module dynamically adjusts the segment size based on data size and type to optimize network resource utilization and transmission efficiency. This process is implemented through a segmentation strategy engine, which automatically calculates the optimal segment size based on predefined rules and real-time data characteristics. The specific steps are as follows: Obtain the data size label and type label of the data; First, the input data is categorized. Data types include, but are not limited to, transaction data, log data, video stream data, and batch file data. Categorization is based on data metadata (such as file extensions and content types) or business rules (such as data source). For example, transaction data is categorized as high-priority real-time data, log data as medium-priority batch data, and video stream data as low-priority streaming data.

[0031] For each data segment, the module calculates the initial segment size based on the data size and type. The calculation uses the following formula: ; in, It is the calculated segment size (unit: bytes). This is the total data size (unit: bytes). This refers to data type weights, dynamically allocated based on the data type: transaction data (T=10), log data (T=5), video stream data (T=2), and other data (T=1). Weight values ​​are based on data priority and transmission requirements; higher-priority data uses smaller segments to reduce latency. and It is an adjustment coefficient, obtained through training with historical transmission data, and has a default value. =1024 (scaling factor) =512 (type factor), This is the minimum segment size, set to 64KB by default to ensure efficiency for small data segments. This is the maximum segment size, set to 1MB by default to avoid network congestion. The logarithmic function in the formula handles the non-linear growth of data size, ensuring that the segment size increases slowly with the data size, avoiding latency caused by excessively large segments.

[0032] During data transmission, the module monitors network conditions (such as latency and packet loss rate) and transmission performance (such as throughput) in real time. If network congestion is detected (packet loss rate exceeding 5%), the segment size is dynamically reduced using the formula: ; in, This is the adjusted segment size. This is the current packet loss rate (unit: percentage). This is an adjustment factor, with a default value of 0.1, used to control the adjustment range. Conversely, if network conditions are good (latency less than 50ms), the segment size can be increased appropriately to improve throughput, but not exceeding [a certain value]. .

[0033] Optionally, the segmented encryption and compression process includes: The data is segmented with overlap to obtain n data segments. After segmentation, the metadata of each segment (such as segment number and size) is recorded for subsequent reassembly and verification. For example, data A is divided into 3 data segments: A1, A2, A3, A4, and A5. A1 and A2 share overlapping data in A12, and A2 and A3 share overlapping data in A23. The same applies to other segments.

[0034] The relevant data segments are encrypted based on the overlapping data. Since different data segments may be transmitted through different transmission paths, even if an attacker can crack A2 encryption, it cannot guarantee that they can crack A3. The overlapping data contained in different data segments is highly likely to be different. The data segments can be encrypted by superimposing the characters and order of the overlapping data. The terminal can only decompress and decrypt the original data after receiving all the correct data segments (considering that the compression process may also have errors, it is designed to require decompression and decryption before verification). This can enhance the security of both the transmission process and the end of the transmission.

[0035] Specifically, regarding one of the aforementioned data: If the first and non-terminal data segments are encrypted based on the overlapping data at their corresponding ends, then the terminal data segments are encrypted based on the data specified at the beginning of the first data segment (which may be represented as A01). If the first data segment is not encrypted based on the overlapping data of its corresponding front end, then the first data segment is encrypted based on the specified data at the end of the last data segment (which can be represented as A50). The lengths of the specified beginning data and the specified end data can be consistent with the lengths of the overlapping data.

[0036] In addition, the size of overlapping data can be dynamically adjusted based on data security requirement level, load conditions, and statistical results of data errors or omissions. For example, the security requirement level is positively correlated with the size of overlapping data, the load size is negatively correlated with the size of overlapping data, and the probability of data errors or omissions is negatively correlated with the size of overlapping data, thus achieving a balance between efficiency and security.

[0037] The encrypted data is then compressed in segments; Optionally, the data distribution module is also used to monitor the push volume and trigger a flow control mechanism when it exceeds a threshold.

[0038] Optionally, the transmission optimization module is responsible for dynamically selecting the optimal transmission path and protocol based on real-time network parameters to ensure the efficiency and stability of data transmission. This module is implemented through a path protocol decision engine, which integrates real-time monitoring, evaluation, and selection mechanisms. The specific working process is as follows: The transmission path and protocol are selected by real-time monitoring of network parameters, including: Latency: The round-trip time (RTT) for a data packet from source to destination, measured in milliseconds (ms). Bandwidth: Available network bandwidth, measured in megabits per second (Mbps). Packet Loss Rate: The percentage of data packets lost during transmission, expressed as a percentage (%). Jitter: The amount of change in latency, measured in milliseconds (ms). Network Load: The utilization rate of current network links, expressed as a percentage (%). This data is collected from multiple nodes via lightweight protocols (such as ICMP or custom UDP packets) and stored in a temporary cache (such as Redis) for real-time analysis. The raw network parameters collected are normalized to eliminate dimensional differences. Normalization uses a Min-Max scaling method to transform each parameter to the [0,1] interval.

[0039] Maintain a list of available paths (e.g., multiple paths obtained through BGP or SDN controllers), and calculate a cost score for each path. The cost score is calculated based on the network parameters and their corresponding weights; the lower the score, the better the path. The weights of the network parameters are determined according to the service priority and network characteristics of data transmission. The default configuration weights are: latency (0.3), bandwidth (0.25), packet loss rate (0.2), jitter (0.15), and network load (0.1). The weights can be adjusted according to the actual scenario. For example, in high real-time transmission scenarios, the weights of latency and jitter can be increased, and in high reliability transmission scenarios, the weight of packet loss rate can be increased.

[0040] The cost score is calculated using a common weighted formula, specifically: Cost score = normalized delay value × corresponding delay weight + normalized bandwidth value × corresponding bandwidth weight + normalized packet loss rate value × corresponding packet loss rate weight + normalized jitter value × corresponding jitter weight + normalized network load value × corresponding network load weight. All network parameters are scaled to the [0,1] range using Min-Max. Packet loss rate, jitter, and network load are positively correlated parameters (the larger the value, the higher the cost), latency is a positively correlated parameter, and bandwidth is a negatively correlated parameter (the larger the value, the lower the cost; after normalization, 1 - the normalized value is used in the calculation).

[0041] If the cost difference between multiple paths is less than 5%, a load balancing strategy is adopted to distribute the data flow across multiple paths. The transmission protocol is dynamically selected based on network parameters and data type labels; protocol options include TCP, UDP, and QUIC; each network parameter is scaled to the [0,1] range using a Min-Max algorithm. If the packet loss rate is greater than 0.1 or the jitter is greater than 0.2, select the UDP protocol and enable forward error correction to reduce retransmission delay. If the delay is greater than 0.5 and the bandwidth is less than 0.3, then the QUIC protocol is chosen to take advantage of its multiplexing and fast handshake. In other cases, TCP is chosen to ensure reliability. For specific data types (such as video streams), real-time performance is prioritized; for transaction data, reliability is prioritized.

[0042] Optionally, during data transmission, the module continuously monitors network parameters. If a parameter change exceeds a threshold (e.g., a 20% increase in latency), path and protocol selection is retried. The adjustment frequency adapts to network stability: in volatile networks, the adjustment frequency is increased to once per second; in stable networks, it is reduced to once every 10 seconds.

[0043] In addition, the frequency at which the transmission optimization module adjusts the transmission path and protocol can also be positively correlated with the variance of the delay within a specified time period in the past. It can calculate the variance of the delay within the past minute. If the variance exceeds the threshold of 100ms², the adjustment frequency is increased to once per second; otherwise, it is reduced to once every 10 seconds.

[0044] Optionally, the client feedback adaptive module collects CPU utilization and network bandwidth feedback from the client. When high load is detected, it reduces the number of threads via server-side instructions; simultaneously, it uses a thread pool manager to adjust thread priorities, ensuring that critical data is processed first. Dynamic thread adjustment optimizes resource allocation, avoids resource waste and contention, and improves system response speed and throughput, making it particularly suitable for environments with large load fluctuations.

[0045] Optionally, the data fault tolerance compensation module calculates a checksum after data transmission and compares it with the client's checksum result. It then uses the data segmentation to quickly identify the location of inconsistent data. If inconsistent, it triggers the ARQ protocol to retransmit the lost data segment. The number of retransmissions is dynamically adjusted based on network conditions. This checksum retransmission mechanism ensures data integrity and reliability, reduces manual intervention, and improves the system's fault tolerance and data consistency.

[0046] Optionally, the training of the distribution strategy prediction model specifically includes: The data preparation steps include collecting terminal operating environment information and performing exploratory data analysis and preprocessing; the scheduler collects client feedback data every 5 seconds from the client feedback adaptation module, including data reception rate, error rate, CPU utilization, and memory usage; it collects network status data from the transmission optimization module, including latency, bandwidth, and packet loss rate; and it collects data queue length and current thread count from the system monitoring module. This data is transmitted to the scheduler in real time via a message queue (such as Kafka). The collected raw data undergoes cleaning and normalization. Cleaning includes removing outliers (e.g., using the Z-score method to identify and filter data points exceeding ±3 standard deviations), and normalization uses Min-Max scaling to transform the data to the [0,1] range.

[0047] The model building steps include determining the explanatory variables and estimating the regression coefficients using the least squares method. Specifically: ; in, It is the prediction parameter (number of threads). and These are explanatory variables (such as network latency, data queue length). , and These are regression coefficients, estimated using the least squares method from historical data. This is the error term. During model training, 10-fold cross-validation is used to avoid overfitting, and the adjusted R-squared is calculated to evaluate the goodness of fit.

[0048] The model evaluation steps include calculating the coefficient of determination and performing statistical tests. The model optimization steps involve optimizing the model through variable transformation or feature selection. The model application steps involve using the optimized model to make predictions and evaluate its performance. The distribution strategy prediction model caches the predicted number of threads in the Redis database and dynamically adjusts the local concurrent thread count and the terminal ingestion thread count based on the prediction results.

[0049] This application also discloses a large-scale data transmission method based on an adaptive distributed architecture, which runs on the aforementioned large-scale data transmission system based on an adaptive distributed architecture.

[0050] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A large-scale data transmission system based on an adaptive distributed architecture, characterized in that, include: The data preprocessing module is used to extract data from the data warehouse and push it to the message topic; The data distribution module is used to classify data, perform segmented encryption and compression processing, and then distribute the data according to the specified number of threads. The transmission optimization module is used to select the transmission path and protocol based on network parameters; The client feedback adaptive module is used to dynamically adjust the thread pool size and thread priority based on the client's received status and the server's instructions. The data fault tolerance and compensation module is used to verify and compensate for data loss; The scheduler is used to comprehensively analyze client feedback and network status, and dynamically adjust the specified number of threads using a distribution strategy prediction model.

2. The large-scale data transmission system based on an adaptive distributed architecture according to claim 1, characterized in that, The data distribution module's operation includes: Obtain the data size label and type label of the data; Configure segment sizes according to the data size labels, and prioritize data distribution according to the type labels; The segment size increases with the data size, and the type labels, in order of priority from low to high, include video stream data, log data, and transaction data.

3. The large-scale data transmission system based on an adaptive distributed architecture according to claim 1, characterized in that, The segmented encryption and compression process includes: The data is segmented with overlap to obtain n data segments; The relevant data segments are encrypted based on the overlapping data; The encrypted data is then compressed in segments; Specifically, regarding one of the aforementioned data: If the first and non-terminal data segments are encrypted based on the overlapping data at their corresponding ends, then the terminal data segments are encrypted based on the data specified at the beginning of the first data segment. If the first data segment is not encrypted based on the overlapping data of its corresponding front end, then the first data segment is encrypted based on the specified data at the end of the last data segment. The lengths of the specified beginning data and the specified end data are the same as those of the overlapping data.

4. The large-scale data transmission system based on an adaptive distributed architecture according to claim 1, characterized in that, The data distribution module is also used to monitor the push volume and trigger a flow control mechanism when it exceeds a threshold.

5. The large-scale data transmission system based on an adaptive distributed architecture according to claim 1, characterized in that, The operation of the transmission optimization module includes: The transmission path and protocol are selected by real-time monitoring of network parameters, including latency, bandwidth, packet loss rate, jitter, and network load. Maintain a list of available paths and calculate a cost score for each path. The cost score is calculated based on the network parameters and their corresponding weights. The lower the score, the better the path. If the cost difference between multiple paths is less than 5%, a load balancing strategy is adopted to distribute the data flow across multiple paths. The transport protocol is dynamically selected based on network parameters and data type labels; protocol options include TCP, UDP, and QUIC. If either the packet loss rate or the jitter parameter exceeds the corresponding threshold, the UDP protocol is selected and forward error correction is enabled to reduce retransmission latency. If the delay is greater than its corresponding threshold and the bandwidth is less than its corresponding threshold, then the QUIC protocol is selected to take advantage of its multiplexing and fast handshake. In other cases, choose the TCP protocol to ensure reliability.

6. The large-scale data transmission system based on an adaptive distributed architecture according to claim 1, characterized in that, The frequency at which the transmission optimization module adjusts the transmission path and protocol is positively correlated with the variance of the delay over a specified time period in the past.

7. The large-scale data transmission system based on an adaptive distributed architecture according to claim 1, characterized in that, The client feedback adaptive module collects CPU usage and network bandwidth feedback from the client. When high load is detected, it reduces the number of threads through server-side instructions; at the same time, it uses a thread pool manager to adjust thread priorities to ensure that critical data is processed first.

8. The large-scale data transmission system based on an adaptive distributed architecture according to claim 3, characterized in that, The data fault tolerance compensation module calculates a checksum after data transmission and compares it with the client's checksum result. It quickly identifies the location of inconsistent data by using the data segmentation. If inconsistent, it triggers the ARQ protocol to retransmit the lost data segment. The number of retransmissions is dynamically adjusted based on network conditions.

9. The large-scale data transmission system based on an adaptive distributed architecture according to claim 1, characterized in that, The training of the distribution strategy prediction model specifically includes: The data preparation steps include collecting terminal operating environment information and conducting exploratory data analysis and preprocessing; The model building steps include determining explanatory variables and estimating regression coefficients using the least squares method; The model evaluation steps include calculating the coefficient of determination and performing statistical tests. The model optimization steps involve optimizing the model through variable transformation or feature selection. The model application steps involve using the optimized model to make predictions and evaluate its performance. The distribution strategy prediction model caches the predicted number of threads in the Redis database and dynamically adjusts the local concurrent thread count and the terminal ingestion thread count based on the prediction results.

10. A large-scale data transmission method based on an adaptive distributed architecture, characterized in that, A large-scale data transmission system based on an adaptive distributed architecture as described in any one of claims 1-8.

Images