Mass data access control method for adaptive data access rate control

Abstract

The invention provides a mass data access control method for self-adaptive data access rate control, which comprises the following steps of: setting system parameters such as an initial data access rate, a performance threshold value and a monitoring period, and acquiring key indexes in real time in an access channel; processing the collected key indexes in real time, and storing the key indexes in a distributed database; calculating performance indexes such as processing capability and resource utilization rate of the current system based on the key indexes; the real-time performance indexes are analyzed through distributed calculation, and the current state of the system is evaluated; the real-time performance index analysis result is compared with a preset performance threshold value, and whether the data access rate needs to be adjusted or not is judged; and dynamically adjusting the data access rate to be matched with the real-time performance index according to a comparison result. According to the method, the limitation of a traditional data access mode is effectively solved, the access rate is dynamically adjusted so as to adapt to continuously changing network environments and service requirements, and the flexibility and adaptability of the system are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for controlling mass data access with adaptive data access rate control. Background Art

[0002] Massive scalable data access technology for massive devices, the access capability of industrial heterogeneous IoT platforms is an important indicator for evaluating IoT platforms. Massive data access control technology is mainly used to process real-time access and management of large-scale data streams. With the rapid development of IoT, big data and cloud computing, more and more devices and sensors need to access the network and transmit data, which puts higher requirements on data access control technology. It aims to ensure that data can access the system quickly and stably in an efficient, flexible and scalable manner, while ensuring the performance and resource utilization of the system. Traditional data access control usually uses a fixed access rate, which cannot be dynamically adjusted according to the actual network environment and business needs; in the case of changes in network load or burst traffic, the fixed rate may cause system overload or resource waste, affecting the stability and efficiency of data transmission.

[0003] Prior art 1, application number: CN 202410856073.4 discloses a massive data privacy protection system and method, which performs data encryption and decryption, identity identification and authentication based on the PKI+CPK system, and utilizes the characteristics of CPK identification authentication such as lightweight, fast authentication, strong processing capability, small computing resources, and large scale to achieve local and efficient authentication of massive data. The mandatory access control system is combined with the ciphertext access policy library related to data operation attributes provided by PKI+CPK to achieve on-demand access control of data such as reading, writing, and execution in limited time domains, limited regions, and limited roles. Although the combination of mandatory access control and the efficient identification and authentication of PKI+CPK can be applied to scenarios such as high-efficiency privacy computing and security applications of massive data, it can solve the computing performance bottleneck problem currently faced by privacy protection. However, the adjustment of the access rate is not involved, resulting in the processing capacity of massive data being limited by the fixed access rate, and the processing potential of massive data cannot be further improved.

[0004] Prior art 2, application number: CN202311517981.2 discloses a method, system, device and storage medium for high-speed processing of massive data, by dividing the data table where the target data type is stored into multiple containers, wherein the container stores data and has a paging index, and responds to data processing instructions, directly controlling each container to perform corresponding data processing based on the paging index of each container. Although data can be processed directly through the data container obtained by segmentation, there is no need to deploy a third-party database server, which has more advantages in client applications, and does not need to rely on services, saving performance consumption of communicating with services and data processing efficiency. Dividing the data table into multiple containers can use multi-threading for data processing, which improves the efficiency of data processing. However, the air access rate mainly depends on the data table interface, resulting in a limited rate of accessing data, and it is impossible to further improve the efficiency of data storage.

[0005] Prior art three, application number: CN202310746363.9 discloses a method and system for high-speed access to ORC external tables, the method comprising: obtaining an ORC file to be scanned, constructing a pre-read list of ORC files, and passing the ORC file to be scanned and the constructed pre-read list to the storage layer; using a storage engine to perform pre-read processing on the ORC file to be scanned according to the constructed pre-read list; scanning the ORC file that has been pre-read, and dynamically controlling the speed of the pre-read processing during the scanning process of the ORC file that has been pre-read. Although the method and system for high-speed access to ORC external tables can improve the query performance of ORC external tables and improve query efficiency without increasing the development workload. However, the query efficiency depends on the pre-read speed of the storage engine, and does not involve adaptive adjustment of the interface rate, resulting in a low degree of intelligence in the pre-read processing.

[0006] At present, the existing technologies 1, 2 and 3 have the problem that the access rate of massive data cannot be adjusted dynamically, resulting in low resource utilization and mismatch between data flow and access rate. Therefore, the present invention provides a massive data access control method with adaptive data access rate control, which adaptively adjusts the data access rate according to a preset adjustment strategy. Summary of the invention

[0007] In order to solve the above technical problems, the present invention provides a massive data access control method with adaptive data access rate control, comprising:

[0008] Set the initial data access rate, performance threshold and monitoring period system parameters, collect key indicators of data packet quantity, size and transmission rate in real time in the access channel; process the collected key indicators in real time and store them in a distributed database;

[0009] Based on key indicators, calculate the processing capacity and resource utilization performance indicators of the current system; use distributed computing to analyze real-time performance indicators and evaluate the current status of the system; compare the real-time performance indicator analysis results with the preset performance threshold to determine whether the data access rate needs to be adjusted; based on the comparison results, dynamically adjust the data access rate to match the real-time performance indicators;

[0010] The adjusted access rate is applied to the system to adjust the rate of the data access channel in real time; through the feedback mechanism, the adjusted access rate is fed back to the key indicators in real time to form a closed-loop control.

[0011] Optionally, the process of real-time processing of the collected key indicators includes the following steps:

[0012] The key indicator collection devices are connected to the distributed database to form a distributed collection system, and the key indicator collection devices are grouped according to the collection content corresponding to them; the key indicator collection devices in the same group summarize the key indicators of the same collection cycle to obtain classified key indicators;

[0013] Obtain the number and location of nodes in the distributed database, divide the collected content into several parts according to the key indicators, use the collection period as the timestamp to distribute the divided collected content to different nodes; use the timestamp as the index to establish the query path of the collected content;

[0014] Establish the relationship between the index of the distributed database and the system. When real-time performance indicator analysis is required, the collected content is retrieved from the distributed database according to the index, and the collected content language calculates the processing capacity and resource utilization performance indicators of the current system.

[0015] Optionally, the process of dynamically adjusting the data access rate to match the real-time performance indicator includes the following steps:

[0016] Obtain key indicators from the distributed database according to the index, and filter out the number, size and transmission rate of data packets from the key indicators; parse the key indicators to obtain the number and total data volume of data packets processed per unit time, and evaluate the system throughput; analyze the transmission rate to determine the utilization efficiency of the current network bandwidth, and evaluate the system resource utilization based on the utilization efficiency;

[0017] Through distributed computing, the throughput and resource utilization are analyzed to obtain the real-time performance index analysis results of the current system, and the trend of the real-time performance index analysis results is predicted to predict the changes in throughput and resource utilization in the future.

[0018] Determine whether to start the data access rate adjustment program according to the real-time performance indicator analysis results. If the real-time performance indicator analysis results reach the preset performance threshold, the data access rate adjustment program is started, otherwise, it is not started; determine the adjusted data access rate according to the real-time performance indicator analysis results, and determine the changed data access rate according to the changes.

[0019] Optionally, the process of obtaining the real-time performance indicator analysis results of the current system includes the following steps:

[0020] Use real-time processing programs to perform distributed computing, extract the throughput and resource utilization in the distributed message queue in the distributed computing; send the throughput and resource utilization to the real-time performance indicator analysis engine;

[0021] Pre-calculate the throughput and resource utilization through the real-time performance indicator analysis engine and obtain the corresponding pre-calculation results; save the pre-calculation results corresponding to the throughput and resource utilization in the distributed file system;

[0022] When analyzing the real-time performance indicators, an analysis request is sent to the real-time performance indicator analysis engine, and the real-time performance indicator analysis is performed based on the throughput and resource utilization.

[0023] Optionally, the process of precalculating throughput and resource utilization includes the following steps:

[0024] Configure cache system parameters, including cache size, expiration time, and data sharding strategy, based on system load and data access patterns;

[0025] In the real-time performance indicator analysis engine, pre-calculation logic is defined to perform preliminary calculations based on throughput and resource utilization data to generate intermediate results. The pre-calculated intermediate results are stored in the cache system. Each pre-calculated result corresponds to a cache key-value pair, where the key is the timestamp or identifier of the data and the value is the pre-calculated result.

[0026] When real-time performance indicator analysis is required, the cache system is queried first; if the corresponding pre-calculated results exist in the cache, the cached data is directly used for analysis.

[0027] Optionally, the process of starting the data access rate adjustment procedure comprises the following steps:

[0028] Collecting traffic information of the system storing the data access rate adjustment program interacting with external devices, obtaining configuration information for controlling the data access rate of the system, and generating a data access rate adjustment program for adjusting the data access rate of the system according to the configuration information and traffic information;

[0029] The data access rate adjustment program includes determining the data access rate between the current system and the external device, determining the maximum data throughput of the current data access rate, performing rate control in response to the configuration information representation, and determining whether the maximum data throughput matches the flow information; if not, generating a data access rate adjustment program for adjusting the data access rate between the current system and the external device;

[0030] The data access rate of the system is adjusted based on the access rate adjustment program; when the flow information changes, the external device sends a flow change notification to the system, and the system re-acquires the flow information based on the flow change notification, and generates a data access rate adjustment program for adjusting the data access rate of the system based on the configuration information and the flow information.

[0031] Optionally, the process of generating a data access rate adjustment program comprises the following steps:

[0032] Real-time perception and prediction of traffic information patterns, identification of peak and valley periods of traffic information and sudden traffic events; through a built-in decision engine in the system, decisions are made based on multi-dimensional data analysis results and configuration information;

[0033] Dynamically generate an adaptive data access rate adjustment program based on the decision, and automatically optimize and adjust the data access rate adjustment percentage based on historical adjustment results and current traffic feedback;

[0034] According to the data access rate adjustment procedure and the data access rate adjustment percentage, a dynamic adjustment rate according to real-time traffic changes is generated.

[0035] Optionally, the decision-making process includes the following steps:

[0036] The decision engine loads the preset configuration information to provide basic parameters for decision generation; the decision engine loads the preset decision rule library, including increasing bandwidth during peak traffic periods, reducing bandwidth during trough periods, and strategies for dealing with sudden traffic events;

[0037] Based on the analysis results of current traffic information, the decision engine matches the decision rules from the rule base and optimizes the matched rules in combination with historical adjustment results and current traffic feedback;

[0038] The decision engine generates the final decision plan based on the matching decision rules and optimization results, including the data access rate adjustment percentage and resource allocation strategy, and records the decision process and results.

[0039] Optionally, the process of generating the final decision solution includes the following steps:

[0040] The decision engine monitors the data access rate of the current system in real time and analyzes its fluctuations and trends. It identifies whether the rate deviates from the normal range by comparing historical data with preset performance indicators.

[0041] Based on the analysis results, the decision engine calculates a dynamic adjustment percentage, which includes the current system load, changes in user demand, and network environment factors; the decision engine automatically applies the calculated adjustment percentage to the system to adjust the data access rate in real time;

[0042] According to the adjustment requirements of data access rate, the decision engine adjusts bandwidth resources accordingly; monitors the load of each server, and achieves load balancing by dynamically adjusting task allocation.

[0043] Optionally, the process of forming a closed-loop control includes the following steps:

[0044] The system monitors the impact of the adjusted transmission rate on performance indicators in real time and dynamically adjusts the feedback strategy; the adjusted transmission rate is updated synchronously with key indicator data;

[0045] The system collects new key indicators again and enters the next cycle;

[0046] Repeat the above steps to ensure that the system is always in the best operating state.

[0047] The system initialization and data acquisition of the present invention ensure that the system has basic operating conditions when it starts up and avoids disordered states; through a high-performance traffic monitoring module, key indicators such as the number, size and transmission rate of data packets in the access channel are collected in real time to provide a data basis for subsequent performance analysis; distributed computing and real-time data processing technology are used to process the collected data in real time and store it in a distributed database to ensure the timeliness and availability of the data. Significance: Provide accurate data support for the operation of the system to ensure that the system can be dynamically adjusted based on real-time data; through real-time processing and storage, ensure the efficient use of data and provide a reliable basis for the adaptive adjustment of the system. Performance analysis and rate adjustment, based on the collected key indicators, calculate the performance indicators such as the processing capacity and resource utilization of the current system to provide a quantitative basis for system status evaluation; use distributed computing technology to analyze real-time performance indicators, evaluate the current state of the system, and ensure that the system can respond to changes in the network environment in a timely manner; compare the intelligent algorithm with the preset performance threshold, dynamically adjust the data access rate, and ensure that the system can maintain an efficient operating state under different network environments. Significance: By dynamically adjusting the access rate, the system can adapt to the ever-changing network environment and business needs, significantly improving the flexibility and adaptability of the system; through intelligent algorithms and optimization strategies, the system can allocate resources more reasonably and improve the overall performance and efficiency of the system. The closed-loop control and feedback mechanism applies the adjusted access rate to the system, adjusts the rate of the data access channel in real time, and ensures that the system can respond to adjustment instructions in a timely manner; through the feedback mechanism, the adjusted access rate is fed back to the key indicators in real time to form a closed-loop control to ensure that the system adjustment process is continuous and dynamic. Significance: Through closed-loop control, the system can continuously optimize the access rate, improve the stability and reliability of data transmission, and ensure that the system can still operate stably in a complex network environment; through continuous feedback and adjustment, the system can achieve self-optimization, reduce manual intervention, and improve the automation level of the system.

[0048] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0049] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0051] Figure 1This is a flow chart of a method for controlling access to massive data with adaptive data access rate control in Embodiment 1 of the present invention;

[0052] Figure 2 This is a process diagram of real-time processing of collected key indicators in Example 2 of the present invention;

[0053] Figure 3 This is a process diagram of dynamically adjusting the data access rate to match the real-time performance index in Embodiment 3 of the present invention;

[0054] Figure 4 A process diagram of obtaining the real-time performance index analysis result of the current system in Example 4 of the present invention;

[0055] Figure 5 This is a process diagram for precalculating throughput and resource utilization in Embodiment 5 of the present invention;

[0056] Figure 6 This is a process diagram of starting a data access rate adjustment procedure in Embodiment 6 of the present invention;

[0057] Figure 7 A process diagram of generating a data access rate adjustment program in Embodiment 7 of the present invention;

[0058] Figure 8 A process diagram for generating a decision in Embodiment 8 of the present invention;

[0059] Fig. 9 A process diagram for generating a final decision solution in Embodiment 9 of the present invention;

[0060] Fig.10 This is a process diagram for forming a closed-loop control in Example 10 of the present invention. DETAILED DESCRIPTION

[0061] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0062] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items.

[0063] When the following description relates to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are only examples of devices and methods consistent with some aspects of the present application. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0064] Example 1: Figure 1 As shown, an embodiment of the present invention provides a method for controlling massive data access with adaptive data access rate control, comprising the following steps:

[0065] S100: Set system parameters such as the initial data access rate, performance threshold and monitoring period, collect key indicators such as the number, size and transmission rate of data packets in real time in the access channel; process the collected key indicators in real time and store them in a distributed database;

[0066] S200: Based on the key indicators, calculate the performance indicators such as the processing capacity and resource utilization of the current system; use distributed computing to analyze the real-time performance indicators and evaluate the current state of the system; compare the real-time performance indicator analysis results with the preset performance threshold to determine whether the data access rate needs to be adjusted; according to the comparison results, dynamically adjust the data access rate to match the real-time performance indicators;

[0067] S300: Apply the adjusted access rate to the system to adjust the rate of the data access channel in real time; through the feedback mechanism, feed back the adjusted access rate to the key indicators in real time to form a closed-loop control.

[0068] The working principle and beneficial effects of the above technical solution are as follows: In this embodiment, the initial data access rate, performance threshold, monitoring period and other system parameters are first set, and key indicators such as the number, size and transmission rate of data packets are collected in real time in the access channel; the collected key indicators are processed in real time and stored in a distributed database; secondly, based on the key indicators, the processing capacity and resource utilization of the current system (an intelligent platform for processing massive data access, mainly used in scenarios requiring high concurrent data processing, such as the Internet of Things (IoT), big data analysis, cloud computing, etc.) are calculated; the real-time performance indicators are analyzed by distributed computing to evaluate the current state of the system; the real-time performance indicator analysis results are compared with the preset performance threshold to determine whether the data access rate needs to be adjusted; according to the comparison results, the data access rate is dynamically adjusted to match the real-time performance indicator; finally, the adjusted access rate is applied to the system to adjust the rate of the data access channel in real time; through the feedback mechanism, the adjusted access rate is fed back to the key indicator in real time to form a closed-loop control to ensure that the adjustment process of the system is continuous and dynamic. The step S100 of the above scheme is system initialization and data collection, which ensures that the system has basic operating conditions when it starts up and avoids disordered state; through the high-performance traffic monitoring module, the key indicators such as the number, size and transmission rate of data packets in the access channel are collected in real time to provide a data basis for subsequent performance analysis; using distributed computing and real-time data processing technology, the collected data is processed in real time and stored in a distributed database to ensure the timeliness and availability of the data. Significance: Provide accurate data support for the operation of the system to ensure that the system can be dynamically adjusted based on real-time data; through real-time processing and storage, ensure the efficient use of data and provide a reliable basis for the adaptive adjustment of the system. Step S200 performance analysis and rate adjustment, based on the collected key indicators, calculate the performance indicators such as the processing capacity and resource utilization of the current system to provide a quantitative basis for system status evaluation; use distributed computing technology to analyze real-time performance indicators, evaluate the current state of the system, and ensure that the system can respond to changes in the network environment in a timely manner; compare with the preset performance threshold through intelligent algorithms, dynamically adjust the data access rate, and ensure that the system can maintain an efficient operating state in different network environments. Significance: By dynamically adjusting the access rate, the system can adapt to the ever-changing network environment and business needs, significantly improving the flexibility and adaptability of the system; through intelligent algorithms and optimization strategies, the system can allocate resources more reasonably and improve the overall performance and efficiency of the system. Step S300 Closed-loop control and feedback mechanism, apply the adjusted access rate to the system, adjust the rate of the data access channel in real time, and ensure that the system can respond to the adjustment instructions in a timely manner; through the feedback mechanism, the adjusted access rate is fed back to the key indicators in real time, forming a closed-loop control to ensure that the system adjustment process is continuous and dynamic.Significance: Through closed-loop control, the system can continuously optimize the access rate, improve the stability and reliability of data transmission, and ensure that the system can still operate stably in a complex network environment; through continuous feedback and adjustment, the system can achieve self-optimization, reduce manual intervention, and improve the system's automation level.

[0069] In summary, this embodiment realizes real-time monitoring, dynamic adjustment and closed-loop control of the system, and effectively solves the limitations of traditional data access methods. The stability, scalability and reliability of the system are significantly improved, and it has a high degree of technical creativity, which can provide users with more efficient, stable and reliable IoT data access services. This embodiment deploys high-performance traffic monitoring to collect key indicators such as the number, size and transmission rate of data packets in the access channel in real time; using distributed computing and real-time data processing technology, the system can dynamically analyze the current processing capacity and resource utilization, and ensure that high-precision and low-latency performance evaluation can be maintained in a complex network environment. Based on the real-time performance analysis results, the system uses an intelligent algorithm to compare with the preset performance threshold to automatically determine whether the data access rate needs to be adjusted; this strategy dynamically adjusts the access rate through machine learning models and optimization algorithms to adapt to the ever-changing network environment and business needs, significantly improving the flexibility and adaptability of the system. After the rate adjustment decision is made, the system feeds back the adjusted access rate to the monitoring module in real time through a feedback mechanism to form a closed-loop control; the feedback mechanism ensures that the system adjustment process is continuous and dynamic, and by continuously optimizing the access rate, the stability and reliability of data transmission are improved, and efficient system self-optimization is achieved. The application of adaptive rate adjustment strategy enables the system to dynamically expand access capabilities according to actual needs when facing massive devices and large-scale data access, avoiding the bottleneck problem of traditional fixed-rate access methods; through distributed architecture and elastic resource management technology, the system can effectively build larger-scale clusters to meet the growing business needs of users. Through automated management and adaptive adjustment strategies, the system can automatically optimize performance during operation and reduce the need for manual intervention; automated management methods not only improve the operating efficiency of the system, but also significantly reduce the cost of cluster maintenance, providing users with more stable and reliable IoT data access services.

[0070] Example 2: Figure 2 As shown, based on Example 1, the process of real-time processing of collected key indicators provided by the embodiment of the present invention includes the following steps:

[0071] S101: Connect key indicator collection devices to a distributed database to form a distributed collection system, and group the key indicator collection devices according to the collection contents corresponding to the key indicator collection devices; the key indicator collection devices in the same group summarize the key indicators of the same collection period to obtain classified key indicators;

[0072] Among them, the equation model for improving the efficiency of key indicator collection equipment accessing the distributed database is:

[0073]

[0074] In the formula, E represents efficiency, which indicates the overall efficiency of key indicator collection devices accessing the distributed database, T represents total time, which indicates the total time of the entire collection process, n represents the number of key indicator collection devices, and D i represents the data volume of the ith device, represents the data volume collected by the ith device, C i Indicates the compression rate of the ith device, which indicates the ratio of the data compressed by the ith device, B i represents the bandwidth of the ith device, represents the data transmission bandwidth of the ith device, P i represents the processing time of the ith device, represents the time required for the ith device to process data, L i represents the load balancing coefficient of the i-th device, indicating the load balancing situation of the i-th device, R represents the network delay, indicating the delay time of data transmission in the network, F represents the network topology optimization coefficient, indicating the optimization degree of the network topology structure, N represents the number of nodes, indicating the number of nodes in the distributed database, S represents the data synchronization time, indicating the time for data synchronization between each node, H represents the data synchronization optimization coefficient, indicating the optimization degree of the data synchronization process, and M represents the number of synchronization nodes, indicating the number of nodes participating in data synchronization; by adjusting various parameters, such as increasing bandwidth, improving device processing capabilities, optimizing network structure, improving data compression and synchronization mechanisms, etc., to maximize efficiency E, thereby improving the efficiency of key indicator collection devices accessing distributed databases;

[0075] S102: Obtain the number and location of nodes in the distributed database, divide the collected content into several parts according to the key indicators, use the collection period as a timestamp to distribute the divided collected content to different nodes; use the timestamp as an index to establish a query path for the collected content;

[0076] S103: Establishing the correlation between the index of the distributed database and the system. When real-time performance index analysis is required, the collected content is retrieved from the distributed database according to the index, and the collected content language calculates the performance indexes such as the processing capacity and resource utilization rate of the current system.

[0077] The working principle and beneficial effects of the above technical solution are as follows: In this embodiment, the key indicator collection device is first connected to the distributed database to form a distributed collection system, and the collection content corresponding to the key indicator collection device is grouped; the key indicator collection devices in the same group summarize the key indicators of the same collection period to obtain the classified key indicators; secondly, the number and position of nodes in the distributed database are obtained, and the collection content of the key indicators is divided into several parts, and the collection period is used as a timestamp to distribute the divided collection content to different nodes; the timestamp is used as an index to establish a query path for the collection content; finally, the index of the distributed database and the system are established. When real-time performance indicator analysis is required, the collection content is taken out from the distributed database according to the index, and the collection content language calculates the processing capacity and resource utilization of the current system and other performance indicators. Step S101 of the above solution: Establish a distributed collection system, by connecting the key indicator collection device to the distributed database, efficient data collection and aggregation can be achieved to ensure the real-time and accuracy of the data; the key indicator collection devices in the same group summarize the key indicators of the same collection period, which is helpful for classification management and subsequent analysis, and reduces data redundancy. Significance: By classifying and summarizing, scattered data can be integrated together to facilitate subsequent analysis and processing; the distributed collection system can significantly improve the efficiency of data collection and reduce the time of data transmission and processing. Step S102: Divide the collected content and distribute it to different nodes. Dividing and distributing the collected content to different nodes can achieve load balancing and avoid overloading of a single node; using the timestamp as an index, the collected content of a specific time period can be quickly located and queried to improve query efficiency. Significance: By distributing data to different nodes, the fault tolerance and scalability of the system can be improved, and the stability and efficiency of data processing can be guaranteed; the establishment of the timestamp index makes data query faster and convenient for real-time monitoring and analysis. Step S103: Establish the association between the index of the distributed database and the system. By taking the collected content out of the distributed database through the index, real-time performance index analysis can be quickly performed; the collected content language calculates the processing capacity and resource utilization of the current system and other performance indicators, which is helpful for real-time monitoring of the system status. Significance: Real-time analysis and resource monitoring can help system administrators to timely discover and solve performance bottlenecks and optimize system operation; through real-time analysis, data support can be provided for decision-making to improve the scientificity and accuracy of decision-making.

[0078] In summary, the real-time processing process of this embodiment not only realizes efficient data collection and processing, but also provides strong support for real-time monitoring and performance optimization of the system, ensuring that the real-time processing of key indicators can meet the requirements of high efficiency, high reliability and high scalability.

[0079] Example 3: Figure 3As shown, based on Example 1, the process of dynamically adjusting the data access rate to match the real-time performance index provided by the embodiment of the present invention includes the following steps:

[0080] S201: Obtain key indicators from a distributed database according to an index, and filter out the number, size, and transmission rate of data packets from the key indicators; parse the key indicators to obtain the number and total data volume of data packets processed per unit time, and evaluate the throughput of the system; analyze the transmission rate to determine the utilization efficiency of the current network bandwidth, and evaluate the resource utilization of the system according to the utilization efficiency;

[0081] S202: Analyze the throughput and resource utilization through distributed computing to obtain the real-time performance indicator analysis results of the current system, make trend predictions on the real-time performance indicator analysis results, and predict changes in the throughput and resource utilization in the future;

[0082] S203: Determine whether to start the data access rate adjustment program according to the real-time performance indicator analysis result. If the real-time performance indicator analysis result reaches the preset performance threshold, the data access rate adjustment program is started, otherwise, it is not started; determine the adjusted data access rate according to the real-time performance indicator analysis result, and determine the changed data access rate according to the change.

[0083] The working principle and beneficial effects of the above technical solution are as follows: In this embodiment, first, key indicators are obtained from a distributed database according to an index, and the number, size and transmission rate of data packets are screened from the key indicators; the key indicators are analyzed to obtain the number of data packets processed per unit time and the total data volume, and the throughput of the system is evaluated; the transmission rate is analyzed to determine the utilization efficiency of the current network broadband, and the resource utilization of the system is evaluated according to the utilization efficiency; secondly, the throughput and resource utilization are analyzed through distributed computing to obtain the real-time performance indicator analysis results of the current system, and a trend forecast is made for the real-time performance indicator analysis results to predict the changes in the throughput and resource utilization in the future; finally, whether to start the data access rate adjustment program is determined according to the real-time performance indicator analysis results. If the real-time performance indicator analysis results reach the preset performance threshold, the data access rate adjustment program is started, otherwise, it is not started; the adjusted data access rate is determined according to the real-time performance indicator analysis results, and the changed data access rate is determined according to the changes. Step S201 of the above scheme obtains and analyzes key indicators. By extracting key indicators such as the number, size and transmission rate of data packets from a distributed database, the system can monitor the current data processing capacity and network utilization in real time. After analyzing the key indicators, the system can evaluate the throughput and resource utilization per unit time. It provides basic data for performance analysis and adjustment, ensuring that the system can make decisions based on actual operation conditions, rather than relying on static or preset parameters. Step S202 analyzes real-time performance indicators and performs trend prediction. Through distributed computing, the system can conduct in-depth analysis of throughput and resource utilization, and predict future trends of these indicators. The prediction capability enables the system to take preventive measures before problems occur. Significance: Trend prediction helps the system identify potential performance bottlenecks or resource over-utilization in advance, thereby avoiding sudden performance degradation or resource waste, and ensuring the stability and efficiency of system operation. Step S203 adjusts the data access rate according to the analysis results. According to the analysis results of the real-time performance indicators, the system can intelligently decide whether the data access rate needs to be adjusted. When the performance indicator reaches the preset threshold, the system will automatically start the adjustment program to optimize the data access rate to match the current system performance. Significance: The dynamic adjustment mechanism ensures that the system can maintain optimal performance under different loads, avoiding performance degradation or resource waste caused by improper data access rate, and improving the overall efficiency of the system and user experience.

[0084] In summary, this embodiment ensures that the system can maintain high efficiency and stability under various operating conditions through real-time monitoring, analysis and adjustment, thereby maximizing resource utilization and system performance.

[0085] Example 4: Figure 4As shown, based on Example 3, the process of obtaining the real-time performance indicator analysis result of the current system provided by the embodiment of the present invention includes the following steps:

[0086] S2021: using a real-time processing program to perform distributed computing, extracting throughput and resource utilization in a distributed message queue in the distributed computing; and sending the throughput and resource utilization to a real-time performance indicator analysis engine;

[0087] S2022: pre-calculate the throughput and resource utilization through the real-time performance indicator analysis engine, and obtain corresponding pre-calculation results; save the pre-calculation results corresponding to the throughput and resource utilization in a distributed file system;

[0088] S2023: When analyzing the real-time performance indicator, an analysis request is sent to the real-time performance indicator analysis engine to perform real-time performance indicator analysis based on throughput and resource utilization.

[0089] The working principle and beneficial effects of the above technical solution are as follows: This embodiment first uses a real-time processing program to perform distributed computing, extracts the throughput and resource utilization in the distributed message queue in the distributed computing; sends the throughput and resource utilization to the real-time performance indicator analysis engine; secondly, the throughput and resource utilization are pre-calculated by the real-time performance indicator analysis engine, and the corresponding pre-calculation results are obtained; the pre-calculation results corresponding to the throughput and resource utilization are saved in the distributed file system; finally, when the real-time performance indicators are analyzed, an analysis request is sent to the real-time performance indicator analysis engine, and the real-time performance indicator analysis is performed according to the throughput and resource utilization. Step S2021 of the above scheme is distributed computing and data extraction. The use of distributed computing can effectively improve the speed and efficiency of data processing, especially when processing large-scale data, and can make full use of the computing power of multiple computers; Throughput and resource utilization extraction Through distributed message queues, the system can monitor and extract the throughput and resource utilization of data processing in real time. The indicators are key data for evaluating system performance. Significance achieved: Through the real-time processing program, the system can obtain and analyze key performance indicators in a timely manner, helping operation and maintenance personnel to quickly understand the operating status of the system; the combination of distributed computing and message queues ensures the accuracy and consistency of data, avoiding the risk of single point failure and data loss. Step S2022 Pre-calculation and data storage: Through the real-time performance indicator analysis engine, the throughput and resource utilization are pre-calculated, and some complex computing tasks can be processed in advance to reduce the computing pressure during real-time analysis; the distributed file system storage saves the pre-calculation results to the distributed file system to ensure high availability and scalability of the data, which is convenient for subsequent rapid access and analysis. Significance achieved: Pre-calculation reduces the computing burden during real-time analysis and improves the response speed and processing efficiency of the system; the use of data persistence distributed file system ensures long-term storage and backup of data, avoids the risk of data loss, and also facilitates data tracing and analysis. Step S2023 Real-time performance indicator analysis, real-time analysis request When the real-time performance indicator needs to be analyzed, the system can respond quickly and analyze it to ensure the real-time and accuracy of the data; dynamically adjust the real-time data based on the throughput and resource utilization, and the system can dynamically adjust resource allocation and task scheduling to optimize system performance. Significance achieved: Real-time performance indicator analysis provides operation and maintenance personnel with immediate decision support, helping to quickly discover and solve problems and ensure the stable operation of the system; system optimization, through the analysis of real-time data, the system can continuously optimize its own performance and resource utilization, improve overall operating efficiency and user satisfaction.

[0090] In this embodiment, step S2021 uses a real-time processing program to perform distributed computing:

[0091]

[0092] In the formula, T represents throughput, which indicates the number of messages processed per unit time, U represents resource utilization, which indicates the utilization of system resources, m represents the number of distributed computing nodes, and D j represents the amount of data of the jth node, represents the amount of data processed by the jth node, and B j represents the bandwidth of the jth node, represents the data transmission bandwidth of the jth node, L j represents the load balancing coefficient of the jth node, represents the load balancing situation of the jth node, P j represents the processing time of the jth node, represents the time required for the jth node to process data, C j represents the computing power of the jth node, represents the computing power of the jth node, R j Indicates the resource utilization of the jth node, indicating the resource utilization of the jth node;

[0093] Step S2022 performs pre-calculation through the real-time performance indicator analysis engine:

[0094]

[0095]

[0096] Where, T pre represents the pre-calculated throughput, represents the pre-calculated throughput, U pre Indicates the pre-calculated resource utilization, indicating the pre-calculated resource utilization;

[0097] Step S2023: Real-time performance indicator analysis:

[0098]

[0099] Where, T real represents the real-time throughput, represents the throughput obtained by real-time analysis, and U real represents the real-time resource utilization, which represents the resource utilization obtained by real-time analysis; α represents the throughput attenuation coefficient, which represents the degree to which the throughput decays over time; β represents the throughput growth coefficient, which represents the degree to which the throughput grows over time; γ represents the resource utilization attenuation coefficient, which represents the degree to which the resource utilization decays over time; δ represents the resource utilization growth coefficient, which represents the degree to which the resource utilization grows over time;

[0100] This embodiment achieves comprehensive monitoring and optimization of system performance through distributed computing, pre-computation and real-time analysis, which not only improves the operating efficiency and stability of the system, but also provides powerful data support and decision-making basis for operation and maintenance personnel, ensuring that the system can operate efficiently and stably.

[0101] Example 5: Figure 5 As shown, based on Example 4, the process of precalculating throughput and resource utilization provided by the embodiment of the present invention includes the following steps:

[0102] S20221: Configure cache system parameters, including cache size, expiration time, and data sharding strategy, based on system load and data access mode;

[0103] S20222: In the real-time performance indicator analysis engine, define pre-calculation logic, perform preliminary calculations based on throughput and resource utilization data, and generate intermediate results; store the pre-calculated intermediate results in a cache system; each pre-calculated result corresponds to a cache key-value pair, where the key is a timestamp or identifier of the data, and the value is the pre-calculated result;

[0104] S20223: When real-time performance indicator analysis is required, the cache system is queried first; if the corresponding pre-calculated results exist in the cache, the cache data is directly used for analysis.

[0105] The working principle and beneficial effects of the above technical solution are as follows: First, according to the system load and data access mode, the present embodiment configures the parameters of the cache system, including cache size, expiration time and data sharding strategy, etc.; secondly, in the real-time performance index analysis engine, the pre-calculation logic is defined, and preliminary calculations are performed based on the throughput and resource utilization data to generate intermediate results; the pre-calculated intermediate results are stored in the cache system; each pre-calculated result corresponds to a cache key-value pair, the key is the timestamp or identifier of the data, and the value is the pre-calculated result; finally, when real-time performance index analysis is required, the cache system is first queried; if the corresponding pre-calculated result exists in the cache, the cache data is directly used for analysis. Step S20221 of the above scheme is to configure the cache system parameters. By reasonably configuring the cache size, it can be ensured that the cache system can store enough pre-calculated results to avoid frequent cache failures and data reloading; setting a suitable expiration time can find a balance between data timeliness and cache utilization to avoid expired data occupying cache space; through the data sharding strategy, data can be distributed to different cache nodes to improve the concurrent processing capability and scalability of the cache. Significance: Reasonable cache configuration can significantly improve the response speed and throughput of the system and reduce the pressure on the back-end storage system; by optimizing cache parameters, system resources can be used more efficiently to improve overall performance and stability. Step S20222 Pre-calculation logic and cache storage: By defining efficient pre-calculation logic, preliminary processing can be performed before the data enters the cache, reducing the complexity and delay of real-time calculations; storing the pre-calculation results in the cache system can quickly access and use these results, avoid repeated calculations, and improve analysis efficiency. Significance: Cache storage of pre-calculation results can significantly reduce the calculation delay of real-time analysis, improve the real-time and response speed of the system; by caching pre-calculation results, repeated calculations can be avoided and the overall efficiency of real-time performance indicator analysis can be improved. Step S20223 Cache access and data utilization: By querying the cache system first, pre-calculation results can be quickly obtained, reducing the frequency of access to the back-end storage system; if the corresponding pre-calculation results exist in the cache, these data can be directly used for real-time performance indicator analysis, reducing the delay and complexity of data processing. Significance: Cache access can significantly improve the system's response speed and ensure the timeliness and accuracy of real-time performance indicator analysis; reducing the frequency of access to the back-end storage system can reduce the overall system load and improve the system's stability and scalability.

[0106] In summary, this embodiment uses cache technology to pre-calculate throughput and resource utilization, which can significantly improve the performance and efficiency of the system. Specifically, reasonable cache configuration can optimize resource utilization, pre-calculation logic and cache storage can reduce computing delays, and cache access can improve the response speed of the system and reduce load. This enables the system to perform real-time performance indicator analysis more efficiently and stably, meeting the application requirements of high concurrency and low latency.

[0107] Example 6: Figure 6 As shown, based on Example 3, the process of starting the data access rate adjustment program provided by the embodiment of the present invention includes the following steps:

[0108] S2031: collecting traffic information of the system storing the data access rate adjustment program interacting with the external device, obtaining configuration information for controlling the data access rate of the system, and generating a data access rate adjustment program for adjusting the data access rate of the system according to the configuration information and the traffic information;

[0109] S2032: wherein the data access rate adjustment program includes determining the data access rate between the current system and the external device, determining the maximum data throughput of the current data access rate, performing rate control in response to the configuration information representation, and determining whether the maximum data throughput matches the flow information; if not, generating a data access rate adjustment program for adjusting the data access rate between the current system and the external device;

[0110] S2033: Adjust the data access rate of the system based on the access rate adjustment program; when the traffic information changes, the external device sends a traffic change notification to the system, and the system re-acquires the traffic information based on the traffic change notification, and generates a data access rate adjustment program for adjusting the data access rate of the system based on the configuration information and the traffic information.

[0111] The working principle and beneficial effects of the above technical solution are as follows: In this embodiment, the flow information of the system storing the data access rate adjustment program interacting with the external device is first collected, the configuration information for controlling the data access rate of the system is obtained, and the data access rate adjustment program for adjusting the data access rate of the system is generated according to the configuration information and the flow information; secondly, the data access rate adjustment program includes determining the data access rate of the current system and the external device, determining the maximum data throughput of the current data access speed record, performing rate control in response to the configuration information representation, and judging whether the maximum data throughput matches the flow information; if not, a data access rate adjustment program for adjusting the data access rate of the current system and the external device is generated; finally, the data access rate of the system is adjusted based on the access rate adjustment program; when the flow information changes, the external device sends a flow change notification to the system, and the system changes according to the flow change notification; the flow information is re-acquired according to the flow change notification, and the data access rate adjustment program for adjusting the data access rate of the system is generated according to the configuration information and the flow information. Step S2031 of the above scheme collects the flow information interacting with external devices, so that the system can monitor the actual situation of data transmission in real time, ensure that the adjustment of data access rate is based on the current network environment and device status; obtain the system's data access rate control configuration information, so that the system can adjust the rate according to preset policies and rules to ensure the efficiency and stability of data transmission; generate a data access rate adjustment program based on flow information and configuration information to ensure that the adjustment process is automated and intelligent, and reduce manual intervention. Significance achieved: By collecting and analyzing flow information in real time, the system can quickly respond to changes in the network environment, ensure that the data access rate is always in the best state, and improve the efficiency and stability of data transmission; automatically generate a data access rate adjustment program, reduce the workload of manual configuration and adjustment, and improve the management efficiency and response speed of the system. Step S2032 determines the data access rate between the current system and the external device to ensure that the system can accurately grasp the current transmission status; determines the maximum data throughput of the current data access rate to provide a reference for rate adjustment; by judging whether the maximum data throughput matches the flow information, the system can identify whether the current rate is suitable for the current network environment, thereby determining whether adjustment is required; if not, a new data access rate adjustment program is generated to ensure that the system can adjust the rate in time to adapt to changes. Significance achieved: By dynamically judging and adjusting the data access rate, the system can adapt to different network environments and device states, ensuring the stability and efficiency of data transmission; through matching judgment and adjustment, the system can optimize resource utilization, avoid resource waste, and improve the overall system operation efficiency.Step S2033: Based on the generated access rate adjustment program, the system can adjust the data access rate in real time to ensure the efficiency and stability of data transmission; the external device sends a traffic change notification to the system, so that the system can timely learn about the changes in traffic information and make corresponding adjustments; the traffic information is re-learned according to the traffic change notification, and a new data access rate adjustment program is generated to ensure that the system can adapt to the new network environment. Significance achieved: Through traffic change notifications and dynamic adjustments, the system can respond to changes in the network environment in real time to ensure the continuity and stability of data transmission; by continuously regenerating and adjusting the data access rate adjustment program, the system can continuously optimize the efficiency of data transmission and resource utilization, and improve the performance and reliability of the overall system.

[0112] In summary, this embodiment ensures the optimization of data access rate and the stability of the system through real-time monitoring, dynamic adjustment and automatic management, thereby improving the efficiency and reliability of data transmission.

[0113] Example 7: Figure 7 As shown, based on Example 6, the process of generating a data access rate adjustment program provided in an embodiment of the present invention includes the following steps:

[0114] S20311: Real-time perception and prediction of traffic information patterns, identification of peak and valley periods of traffic information and sudden traffic events; generation of decisions based on multi-dimensional data analysis results and configuration information through a decision engine built into the system;

[0115] S20312: dynamically generate an adaptive data access rate adjustment program according to the decision, and automatically optimize and adjust the data access rate adjustment percentage according to historical adjustment results and current traffic feedback;

[0116] S20313: Generate a dynamic adjustment rate according to real-time traffic changes based on the data access rate adjustment program and the data access rate adjustment percentage.

[0117] The working principle and beneficial effects of the above technical solution are as follows: the present embodiment firstly perceives and predicts the flow information pattern in real time, identifies the peak period, trough period and sudden flow events of the flow information; generates a decision based on the multi-dimensional data analysis results and configuration information through a decision engine built into the system; secondly, generates an adaptive data access rate adjustment program dynamically according to the decision, automatically optimizes and adjusts the data access rate adjustment percentage according to the historical adjustment results and the current flow feedback; finally, generates a dynamic adjustment rate according to the real-time flow change according to the data access rate adjustment program and the data access rate adjustment percentage. Step S20311 of the above solution is flow information perception and prediction. By real-time collection and analysis of flow data, the system can accurately identify the peak period, trough period and sudden flow events of the flow. This enables the system to predict flow changes in advance and provide data support for subsequent decisions; the decision engine built into the system can comprehensively consider multiple data dimensions (such as user behavior, network conditions, historical data, etc.), conduct in-depth analysis, and thus generate more accurate decisions. Significance: By accurately predicting traffic changes, the system can increase resources in advance during peak hours to avoid network congestion; reduce resource occupancy during trough hours to improve resource utilization; real-time perception and prediction of traffic changes can effectively reduce network delays and freezes, and improve the user's Internet experience. Step S20312 Adaptive data access rate adjustment: According to the generated decision, the system can dynamically generate an adaptive data access rate adjustment program, and can automatically optimize and adjust the data access rate adjustment percentage according to historical adjustment results and current traffic feedback; the system can automatically optimize and adjust the data access rate adjustment percentage according to historical adjustment results and current traffic feedback to ensure the accuracy and effectiveness of the adjustment. Significance: The adaptive data access rate adjustment program can flexibly adjust the access rate according to real-time traffic changes to ensure the reasonable allocation of network resources; by automatically optimizing the adjustment percentage, the system can continuously optimize the adjustment strategy to improve network performance and user experience. Step S20313 Dynamic rate adjustment: According to the generated data access rate adjustment program and adjustment percentage, the system can adjust the data access rate in real time to ensure the dynamic allocation of network resources; the dynamic rate adjustment can flexibly adjust the access rate according to real-time traffic changes to adapt to various complex network environments. Significance: By adjusting the data access rate in real time, the system can efficiently utilize network resources, avoid resource waste and network congestion; dynamic rate adjustment can ensure the stability of network performance and maintain a good network experience even during peak traffic periods.

[0118] In summary, this embodiment can realize intelligent adjustment of data access rate, thereby optimizing the allocation of network resources and improving user experience. The intelligent adjustment mechanism not only improves the stability and performance of the network, but also provides a solid technical foundation for future network optimization.

[0119] Example 8: Figure 8 As shown, based on Example 7, the process of generating a decision provided by the embodiment of the present invention includes the following steps:

[0120] S203111: The decision engine loads the preset configuration information to provide basic parameters for decision generation; the decision engine loads the preset decision rule library, including increasing bandwidth during peak traffic periods, reducing bandwidth during valley periods, and strategies for dealing with sudden traffic events;

[0121] S203112: Based on the analysis results of the current traffic information, the decision engine matches the decision rules from the rule base and optimizes the matched rules in combination with the historical adjustment results and the current traffic feedback;

[0122] S203113: The decision engine generates a final decision plan based on the matching decision rules and optimization results, including the data access rate adjustment percentage and resource allocation strategy, and records the decision process and results.

[0123] The working principle and beneficial effects of the above technical solution are as follows: In this embodiment, the decision engine first loads the preset configuration information to provide basic parameters for decision generation; the decision engine loads the preset decision rule library, including increasing bandwidth during peak traffic periods, reducing bandwidth during trough periods, and strategies for dealing with sudden traffic events; secondly, according to the analysis results of the current traffic information, the decision engine matches the decision rules from the rule library, and optimizes the matched rules in combination with the historical adjustment results and the current traffic feedback; finally, the decision engine generates the final decision plan according to the matched decision rules and the optimization results, including the data access rate adjustment percentage and resource allocation strategy, and records the decision process and results. Step S203111 of the above solution loads the configuration information and the decision rule library. By loading the preset configuration information, the decision engine can obtain the basic parameters required for the decision, ensuring the rationality and feasibility of the decision; loading the preset decision rule library (such as increasing bandwidth during peak traffic periods, reducing bandwidth during trough periods, and strategies for dealing with sudden traffic events, etc.), provides the decision engine with a wealth of decision basis, ensuring that the appropriate decision rules can be quickly matched in different scenarios. Significance achieved: Loading configuration information ensures that the decision engine has clear basic parameters and constraints when generating decisions, avoiding blindness and arbitrariness in decision-making; loading the decision rule library enriches the decision basis of the decision engine, enabling it to cope with a variety of complex scenarios and improve the comprehensiveness and flexibility of decision-making. Step S203112 matches and optimizes decision rules. According to the current traffic information analysis results, the decision engine matches the most appropriate decision rules from the rule library to ensure the real-time and pertinence of the decision; combined with historical adjustment results and current traffic feedback, the matching rules are optimized to improve the accuracy and adaptability of the decision. Significance achieved: By matching decision rules in real time, the decision engine can quickly respond to traffic changes and ensure timely allocation of network resources; by optimizing based on historical adjustment results and current traffic feedback, the decision engine can adaptively adjust decision rules to improve the accuracy and flexibility of decisions and avoid resource waste and performance bottlenecks. Step S203113 generates and records the final decision plan. According to the matching decision rules and optimization results, the decision engine generates the final decision plan, including the data access rate adjustment percentage and resource allocation strategy, to ensure the clarity and executability of the decision; record the decision process and results to facilitate subsequent analysis and optimization, and improve the traceability and transparency of the decision. Significance achieved: Generate a clear decision plan to ensure that the execution of the decision has a basis to rely on, and avoid confusion and deviation in the execution process; by recording the decision process and results, the system can achieve the traceability of the decision, facilitate subsequent analysis and optimization, and improve the scientificity and rationality of the decision.

[0124] In summary, the decision engine of this embodiment can generate reasonable decision solutions comprehensively, intelligently and in real time to ensure the optimal configuration and efficient use of network resources. Each step together enhances the decision-making ability of the decision engine, enabling it to provide scientific, reasonable and efficient decision support in a complex and changing network environment.

[0125] Example 9: Fig. 9 As shown, based on Example 8, the process of generating the final decision solution provided by the embodiment of the present invention includes the following steps:

[0126] S2031131: The decision engine monitors the data access rate of the current system in real time and analyzes its fluctuations and trends. It identifies whether the rate deviates from the normal range by comparing historical data with preset performance indicators.

[0127] S2031132: Based on the analysis results, the decision engine calculates a dynamic adjustment percentage, which includes factors such as the current system load, changes in user demand, and network environment; the decision engine automatically applies the calculated adjustment percentage to the system to adjust the data access rate in real time;

[0128] S2031133: Based on the adjustment requirements of data access rate, the decision engine adjusts bandwidth resources accordingly; monitors the load of each server, and achieves load balancing by dynamically adjusting task allocation.

[0129] The working principle and beneficial effects of the above technical solution are as follows: First, in this embodiment, the decision engine will monitor the data access rate of the current system in real time and analyze its fluctuations and trends; by comparing historical data and preset performance indicators, it can identify whether the rate deviates from the normal range; secondly, based on the analysis results, the decision engine calculates a dynamic adjustment percentage, including factors such as the current system load, user demand changes, and network environment; the decision engine automatically applies the calculated adjustment percentage to the system to adjust the data access rate in real time; finally, according to the adjustment requirements of the data access rate, the decision engine adjusts the bandwidth resources accordingly; monitors the load of each server, and realizes load balancing by dynamically adjusting task allocation. Step S2031131 of the above solution is real-time monitoring and analysis. By real-time monitoring of the data access rate, the system can timely capture the fluctuations and trend changes of the rate to ensure the accuracy and timeliness of the data; by comparing the current data access rate with historical data, it can identify whether the rate deviates from the normal range, helping the system to determine whether it needs to be adjusted; through the preset performance indicators, the system can quantitatively evaluate the current performance status and provide a basis for subsequent adjustments. Significance: Real-time monitoring and analysis ensure that the system can respond to changes in data access rate in a timely manner, avoiding system performance degradation or crash caused by rate fluctuations; through historical data comparison and preset performance indicators, the system can warn of potential problems in advance, perform preventive maintenance, and improve the stability and reliability of the system. Step S2031132 Dynamic adjustment percentage calculation and application: Based on factors such as system load, user demand changes, and network environment, a dynamic adjustment percentage is calculated to ensure the accuracy and adaptability of the adjustment; the calculated adjustment percentage is automatically applied to the system to adjust the data access rate in real time, reduce human intervention, and improve the efficiency and accuracy of the adjustment. Significance: Dynamic adjustment percentage ensures that the system can adjust adaptively according to actual conditions, avoiding performance bottlenecks or resource waste caused by fixed parameters; automated adjustment reduces the time and error of manual operations, and improves the operating efficiency and stability of the system. Step S2031133 Resource allocation strategy: According to the adjustment requirements of the data access rate, bandwidth resources are dynamically adjusted to ensure smooth and efficient data transmission; by monitoring the load conditions of each server, task allocation is dynamically adjusted to achieve load balancing, avoid single point overload, and improve the overall performance of the system. Significance: By dynamically adjusting bandwidth resources and load balancing, the system can optimize resource allocation and ensure efficient operation under high load conditions; the load balancing strategy avoids single point failures, improves system stability and reliability, and ensures that users can continue to receive high-quality services.

[0130] In summary, the decision engine of this embodiment can not only monitor and analyze the operating status of the system in real time, but also dynamically adjust the data access rate and resource allocation to ensure that the system can operate efficiently and stably under various complex situations. Together, the system's performance, stability and user experience are improved, ensuring that the system can cope with various challenges and continue to provide users with high-quality services.

[0131] Example 10: Fig.10 As shown, based on Example 1, the process of forming a closed-loop control provided by the embodiment of the present invention includes the following steps:

[0132] S301: The system monitors the impact of the adjusted transmission rate on the performance index in real time and dynamically adjusts the feedback strategy; the adjusted transmission rate and key index data are updated synchronously;

[0133] S302: The system collects new key indicators again and enters the next cycle;

[0134] S303: Repeat the above steps to ensure that the system is always in the best operating state.

[0135] The working principle and beneficial effects of the above technical solution are as follows: In this embodiment, the system first monitors the influence of the adjusted transmission rate on the performance index in real time and dynamically adjusts the feedback strategy; the adjusted transmission rate is updated synchronously with the key indicator data; secondly, the system collects new key indicators again and enters the next cycle; finally, the above steps are repeated to ensure that the system is always in the best operating state. In the above scheme, step S301 real-time monitoring and dynamic adjustment, the system can instantly capture the performance changes after the transmission rate is adjusted to ensure the real-time and accuracy of the data; according to the results of real-time monitoring, the system can quickly adjust the feedback strategy, optimize the transmission rate, and avoid performance bottlenecks; the adjusted transmission rate is updated synchronously with the key indicator data to ensure the consistency of information in various parts of the system. Significance: Real-time monitoring and dynamic adjustment enable the system to respond quickly to performance changes and reduce delays; through dynamic adjustment, the system can use network resources more efficiently and avoid waste; ensure that network performance is always in the best state and improve the user experience. Step S302 collects new key indicators again. By collecting new key indicators again, the system can obtain the latest performance data and ensure the accuracy of analysis; new data collection provides a basis for the optimization of the next cycle and ensures continuous improvement of the system. Significance: By continuously collecting new data, the system can continuously optimize performance and adapt to the ever-changing network environment; analysis and adjustment based on the latest data make decisions more scientific and accurate. Step S303 repeats the above steps. By repeating the above steps, a closed-loop control system is formed to ensure continuous optimization of performance; the system can automatically adjust and optimize according to real-time data and feedback, and has strong adaptive capabilities. Significance: Closed-loop control ensures that the system can maintain stable operation in various network environments; through continuous optimization and adjustment, the system can provide more reliable services; closed-loop control enables the system to maintain high performance for a long time and provide continuous value to users.

[0136] In summary, the closed-loop control system of this embodiment can monitor in real time, dynamically adjust, and continuously optimize, ensuring that the network performance is always in the best state and providing users with stable, efficient, and reliable services.

[0137] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention belong to the scope of equivalent technologies of the present invention, the present invention is also intended to include these modifications and variations.

Claims

1. A method for controlling mass data access with adaptive data access rate control, characterized in that: Include: Set the initial data access rate, performance threshold and monitoring period system parameters, collect key indicators of data packet quantity, size and transmission rate in real time in the access channel; process the collected key indicators in real time and store them in a distributed database; Based on key indicators, calculate the processing capacity and resource utilization performance indicators of the current system; use distributed computing to analyze real-time performance indicators and evaluate the current status of the system; Compare the real-time performance index analysis results with the preset performance threshold to determine whether the data access rate needs to be adjusted; based on the comparison results, dynamically adjust the data access rate to match the real-time performance index; Apply the adjusted access rate to the system and adjust the rate of the data access channel in real time; Through the feedback mechanism, the adjusted access rate is fed back to the key indicators in real time to form a closed-loop control.

2. The method for controlling mass data access with adaptive data access rate control according to claim 1, characterized in that: The process of real-time processing of the collected key indicators includes the following steps: The key indicator collection devices are connected to the distributed database to form a distributed collection system, and the key indicator collection devices are grouped according to the collection content corresponding to them; the key indicator collection devices in the same group summarize the key indicators of the same collection cycle to obtain classified key indicators; Obtain the number and location of nodes in the distributed database, divide the collected content into several parts according to the key indicators, and use the collection period as the timestamp to distribute the divided collected content to different nodes; Use timestamp as index to establish query path for collected content; Establish the relationship between the index of the distributed database and the system. When real-time performance indicator analysis is required, the collected content is retrieved from the distributed database according to the index, and the collected content language calculates the processing capacity and resource utilization performance indicators of the current system.

3. The method for controlling mass data access with adaptive data access rate control as claimed in claim 1, characterized in that: The process of dynamically adjusting the data access rate to match the real-time performance indicator includes the following steps: Obtain key indicators from the distributed database according to the index, and filter out the number, size and transmission rate of data packets from the key indicators; parse the key indicators to obtain the number and total data volume of data packets processed per unit time, and evaluate the system throughput; analyze the transmission rate to determine the utilization efficiency of the current network bandwidth, and evaluate the system resource utilization based on the utilization efficiency; Through distributed computing, the throughput and resource utilization are analyzed to obtain the real-time performance index analysis results of the current system, and the trend of the real-time performance index analysis results is predicted to predict the changes in throughput and resource utilization in the future. Determine whether to start the data access rate adjustment program according to the real-time performance indicator analysis results. If the real-time performance indicator analysis results reach the preset performance threshold, the data access rate adjustment program is started, otherwise, it is not started; determine the adjusted data access rate according to the real-time performance indicator analysis results, and determine the changed data access rate according to the changes.

4. The method for controlling mass data access with adaptive data access rate control as claimed in claim 3, characterized in that: The process of obtaining the real-time performance indicator analysis results of the current system includes the following steps: Use real-time processing programs to perform distributed computing, extract the throughput and resource utilization in the distributed message queue in the distributed computing; send the throughput and resource utilization to the real-time performance indicator analysis engine; Pre-calculate the throughput and resource utilization through the real-time performance indicator analysis engine and obtain the corresponding pre-calculation results; The pre-calculated results corresponding to the throughput and resource utilization are saved in the distributed file system; When analyzing the real-time performance indicators, an analysis request is sent to the real-time performance indicator analysis engine, and the real-time performance indicator analysis is performed based on the throughput and resource utilization.

5. The method for controlling access to massive data with adaptive data access rate control as claimed in claim 4, characterized in that: The process of precalculating throughput and resource utilization includes the following steps: Configure cache system parameters, including cache size, expiration time, and data sharding strategy, based on system load and data access patterns; In the real-time performance indicator analysis engine, pre-calculation logic is defined to perform preliminary calculations based on throughput and resource utilization data to generate intermediate results; Store the pre-calculated intermediate results in the cache system; each pre-calculated result corresponds to a cache key-value pair, the key is the timestamp or identifier of the data, and the value is the pre-calculated result; When real-time performance indicator analysis is required, the cache system is queried first; if the corresponding pre-calculated results exist in the cache, the cached data is used directly for analysis.

6. The method for controlling access to massive data with adaptive data access rate control as claimed in claim 3, characterized in that: The process of starting the data access rate adjustment procedure includes the following steps: Collecting traffic information of the system storing the data access rate adjustment program interacting with external devices, obtaining configuration information for controlling the data access rate of the system, and generating a data access rate adjustment program for adjusting the data access rate of the system according to the configuration information and traffic information; The data access rate adjustment program includes determining the data access rate between the current system and the external device, determining the maximum data throughput of the current data access rate, performing rate control in response to the configuration information representation, and determining whether the maximum data throughput matches the flow information; if not, generating a data access rate adjustment program for adjusting the data access rate between the current system and the external device; Adjusting the data access rate of the system based on the access rate adjustment procedure; When the flow information changes, the external device sends a flow change notification to the system, and the system re-acquires the flow information according to the flow change notification, and generates a data access rate adjustment program for adjusting the data access rate of the system according to the configuration information and the flow information.

7. The method for controlling mass data access with adaptive data access rate control as claimed in claim 6, characterized in that: The process of generating a data access rate adjustment program includes the following steps: Real-time perception and prediction of traffic information patterns, identification of peak and valley periods of traffic information and sudden traffic events; through a built-in decision engine in the system, decisions are made based on multi-dimensional data analysis results and configuration information; Dynamically generate an adaptive data access rate adjustment program based on the decision, and automatically optimize and adjust the data access rate adjustment percentage based on historical adjustment results and current traffic feedback; According to the data access rate adjustment procedure and the data access rate adjustment percentage, a dynamic adjustment rate according to real-time traffic changes is generated.

8. The method for controlling mass data access with adaptive data access rate control as claimed in claim 7, characterized in that: The decision-making process consists of the following steps: The decision engine loads the preset configuration information to provide basic parameters for decision generation; the decision engine loads the preset decision rule library, including increasing bandwidth during peak traffic periods, reducing bandwidth during trough periods, and strategies for dealing with sudden traffic events; Based on the analysis results of current traffic information, the decision engine matches the decision rules from the rule base and optimizes the matched rules in combination with historical adjustment results and current traffic feedback; The decision engine generates the final decision plan based on the matching decision rules and optimization results, including the data access rate adjustment percentage and resource allocation strategy, and records the decision process and results.

9. The method for controlling access to massive data with adaptive data access rate control as claimed in claim 8, characterized in that: The process of generating the final decision solution includes the following steps: The decision engine monitors the data access rate of the current system in real time and analyzes its fluctuations and trends. It identifies whether the rate deviates from the normal range by comparing historical data with preset performance indicators. Based on the analysis results, the decision engine calculates a dynamic adjustment percentage, which includes the current system load, changes in user demand, and network environment factors; the decision engine automatically applies the calculated adjustment percentage to the system to adjust the data access rate in real time; According to the adjustment requirements of data access rate, the decision engine adjusts bandwidth resources accordingly; monitors the load of each server, and achieves load balancing by dynamically adjusting task allocation.

10. The method for controlling access to massive data with adaptive data access rate control according to claim 1, characterized in that: The process of forming a closed-loop control includes the following steps: The system monitors the impact of the adjusted transmission rate on performance indicators in real time and dynamically adjusts the feedback strategy; the adjusted transmission rate is updated synchronously with key indicator data; The system collects new key indicators again and enters the next cycle; Repeat the above steps to ensure that the system is always in the best operating state.

Images