Composite flow limiting method and system for mass flow data transmission based on Internet of data

Abstract

The invention provides a composite flow limiting method and system for large-flow data transmission based on the Internet of Things, and relates to the technical field of the Internet of Things, and the method comprises the steps: analyzing a protocol header of request data, and determining a target communication protocol of data transmission; determining a target data type and a priority value of the digital object according to metadata in the requested digital object; determining a corresponding first current limiting strategy according to the target communication protocol and the target data type; performing parameter correction on the first current limiting strategy according to the priority value to obtain a second current limiting strategy; and performing current-limiting transmission on the digital object through the second current-limiting strategy. The invention aims to improve the stability of data transmission under a digital networking system.

Description

Technical Field

[0001] This application relates to the field of digital networking technology, and in particular, to a composite current limiting method and system for large flow data transmission based on digital networking. Background Art

[0002] In a digital networking environment, due to problems such as network fluctuations, resource limitations, high concurrency, and data diversity, existing current limiting methods (such as sliding window current limiting, token bucket algorithm, leaky bucket algorithm, etc.) each have limitations and cannot comprehensively and effectively address the challenges brought by large flow data transmission, and are not flexible enough. This may cause the system to overload due to excessive requests, affecting system stability. Summary of the Invention

[0003] In view of this, this application provides a composite current limiting method and system for large flow data transmission based on digital networking, aiming to improve the stability of data transmission in a digital networking system.

[0004] In the first aspect of this application, a composite current limiting method for large flow data transmission based on digital networking is provided. The method includes: Parsing the protocol header of the request data to determine the target communication protocol for data transmission; Determining the target data type and priority value of the digital object according to the metadata in the requested digital object; Determining a corresponding first current limiting strategy according to the target communication protocol and the target data type; Modifying the parameters of the first current limiting strategy according to the priority value to obtain a second current limiting strategy; Performing current limiting transmission on the digital object through the second current limiting strategy.

[0005] Optionally, modifying the parameters of the first current limiting strategy according to the priority value to obtain a second current limiting strategy, including: Determining the target parameter for parameter modification according to the first current limiting strategy; Modifying the target parameter according to the priority value and a preset modification rule to obtain a second current limiting strategy.

[0006] Optionally, modifying the target parameter according to the priority value and a preset modification rule to obtain a second current limiting strategy, including: Determining a basic priority value according to the priority values of all digital objects in the cache queue; Determining the priority deviation value between the priority value of the digital object and the basic priority value; Based on the priority deviation value, the base value and adjustment step of the target parameter, and the target percentage value, parameter correction is performed on the target parameter to obtain a second flow-limiting policy, where the target percentage value is a value corresponding to both the priority value of the digital object and the type of the target parameter.

[0007] Optionally, before parameter correction is performed on the first flow-limiting policy according to the priority value to obtain a second flow-limiting policy, the method further includes: Adjust the priority value of the digital object according to the waiting duration of the digital object in the cache queue; The parameter correction is performed on the first flow-limiting policy according to the priority value to obtain a second flow-limiting policy, including: Parameter correction is performed on the first flow-limiting policy according to the adjusted priority value to obtain a second flow-limiting policy.

[0008] Optionally, before the digital object is flow-limited and transmitted through the second flow-limiting policy, the method further includes: Determine the actual measured value of the network traffic during the data transmission process; According to the actual measured value and the set value of the network traffic, perform dynamic adjustment on the second flow-limiting policy through PID feedback regulation; The flow-limited transmission of the digital object through the second flow-limiting policy includes: Perform flow-limited transmission on the digital object through the second flow-limiting policy after dynamic adjustment.

[0009] Optionally, according to the actual measured value and the set value of the network traffic, performing dynamic adjustment on the second flow-limiting policy through PID feedback regulation includes: In the case where the second flow-limiting policy is the token bucket algorithm, perform dynamic adjustment on the token bucket capacity or token generation rate in the second flow-limiting policy through PID feedback regulation according to the actual measured value and the set value of the network traffic; In the case where the second flow-limiting policy is the leaky bucket algorithm, perform dynamic adjustment on the leaky bucket capacity or water outlet rate in the second flow-limiting policy through PID feedback regulation according to the actual measured value and the set value of the network traffic; In the case where the second flow-limiting policy is the sliding window algorithm, perform dynamic adjustment on the window size or request processing rate in the second flow-limiting policy through PID feedback regulation according to the actual measured value and the set value of the network traffic.

[0010] Optionally, the method further includes: When the priority values of all digital objects in the cache queue are adjusted to the highest priority, average resource allocation is performed for the data transmission of each digital object.

[0011] Optionally, when the target communication protocol is the HTTP protocol, the method further includes: Filter the request data through a regular rule library to determine whether the request data belongs to a known attack pattern; In the case of a rule miss, analyze the parameter semantics of the request data through an AST parsing engine to determine whether the request data is a suspicious request; When the request data is a suspicious request, direct the request data to an isolation environment for execution and perform abnormal behavior monitoring; When no abnormal behavior is detected, respond to the request data.

[0012] Optionally, when the target communication protocol is the gRPC protocol, the method further includes: Verify the RPC method signature in the request data to determine whether the request data is legal. The RPC method signature is a unique signature generated by the server for each RPC method; When the request data is legal, determine whether the call chain of the request data is an abnormal call chain according to the request data and the service dependency graph; When the call chain is an abnormal call chain, block the abnormal call chain. The service dependency graph is a dependency relationship between services constructed based on the information recorded in each data request; or Determine whether the request data is legal according to the dynamic token in the request data. The dynamic token is sent to the client after the server passes the certificate and fingerprint verification of the connection request of the client sending the request data; When the request data is legal, respond to the request data.

[0013] Optionally, when the target communication protocol is the WebSocket protocol, the method further includes: Perform binary magic number verification on the request data to determine whether the request data is legal, and / or perform malicious keyword scanning on the request data to determine whether the request data is legal; When the request data is legal, respond to the request data; Adjust the connection number of each client according to the CPU usage rate of the server; Disconnect the request connection that meets the preset heartbeat monitoring mechanism according to the preset heartbeat monitoring mechanism.

[0014] The second aspect of the present application provides a composite current-limiting system for large-flow data transmission based on the digital networking, and the system includes: A target communication protocol determination module, configured to parse the protocol header of the request data to determine the target communication protocol for data transmission; A metadata parsing module, configured to determine the target data type and priority value of the digital object according to the metadata in the requested digital object; A first current-limiting strategy determination module, configured to determine a corresponding first current-limiting strategy according to the target communication protocol and the target data type; A second current-limiting strategy determination module, configured to correct the parameters of the first current-limiting strategy according to the priority value to obtain a second current-limiting strategy; A current-limiting control module, configured to perform current-limiting transmission on the digital object through the second current-limiting strategy.

[0015] Regarding the prior art, the present application has the following advantages: A composite current-limiting method for large-flow data transmission based on the digital networking provided by the present application. First, the protocol header of the request data is parsed to determine the target communication protocol for data transmission; according to the metadata in the requested digital object, the target data type and priority value of the digital object are determined; according to the target communication protocol and the target data type, a corresponding first current-limiting strategy is determined; according to the priority value, the parameters of the first current-limiting strategy are corrected to obtain a second current-limiting strategy; and current-limiting transmission is performed on the digital object through the second current-limiting strategy. Thus, the present application selects a current-limiting strategy adapted to the current data type and communication protocol from multiple current-limiting strategies to perform current-limiting processing on the digital object by combining multiple current-limiting strategies based on the data type recorded in the metadata of the digital object for data transmission and the communication protocol for data transmission, so as to meet the data transmission requirements in different scenarios, thereby improving the stability and reliability of the system. At the same time, based on the priority level of the digital object, the corresponding current-limiting strategy is adjusted, so as to give a smaller current-limiting degree to the digital object with a higher priority and a larger current-limiting degree to the digital object with a lower priority, thereby optimizing the resource utilization in the data transmission process.

[0016] The above description is only an overview of the technical solution of the present application. In order to be able to understand the technical means of the present application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other objects, features and advantages of the present application more obvious and understandable, the specific embodiments of the present application are specifically exemplified below. Description of the Drawings

[0017] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required in the description of the embodiments or the prior art.

[0018] Figure 1 It is a flowchart of a composite flow limiting method for large - flow data transmission based on the Internet of Data provided by an embodiment of the present application; Figure 2 It is another flowchart of a composite flow limiting method for large - flow data transmission based on the Internet of Data provided by an embodiment of the present application; Figure 3 It is a flowchart of the flow limiting of the token bucket algorithm in a composite flow limiting method for large - flow data transmission based on the Internet of Data provided by an embodiment of the present application; Figure 4 It is a flowchart of the flow limiting of the leaky bucket algorithm in a composite flow limiting method for large - flow data transmission based on the Internet of Data provided by an embodiment of the present application; Figure 5 It is a flowchart of the flow limiting of the sliding window algorithm in a composite flow limiting method for large - flow data transmission based on the Internet of Data provided by an embodiment of the present application; Figure 6 It is a schematic diagram of a composite flow limiting system for large - flow data transmission based on the Internet of Data provided by an embodiment of the present application. Detailed implementation manners

[0019] The following will describe the exemplary embodiments of the present application in more detail with reference to the accompanying drawings.

[0020] For the convenience of understanding the solution, the following explains the proprietary terms: Internet of Data (IOD): It is a virtual data network based on the Internet. Through an open software architecture and standardized protocols, it efficiently connects various data platforms and systems, supports the interconnection, interoperability, and mutual operation of heterogeneous, different - domain, and different - owner data, and forms a data space of "data interconnection, on - demand scheduling, in - domain autonomy, and inter - domain cooperation".

[0021] Digital Object (DO for short): It is a data structure for abstractly describing data resources. It organizes and records the data in a computer system according to basic elements such as identifiers, metadata, and data entities.

[0022] Figure 1 It is a flowchart of a composite flow limiting method for large - flow data transmission based on the Internet of Data provided by an embodiment of the present application. As Figure 1 shown, the method includes: Step S1: Parse the protocol header of the request data to determine the target communication protocol for data transmission.

[0023] In this embodiment, the present application is applied to a digital object network constructed based on the digital object architecture. For the convenience of description, all digital object networks mentioned hereinafter are digital object networks constructed based on the digital object architecture. The data types transmitted under this digital object network are diverse, including sensor data, medical emergency data, text data, video data, etc. Different data types have different focuses during the transmission process, and a single flow-limiting strategy cannot meet the transmission requirements of various different types of data in this digital object network scenario. Therefore, the present application introduces multiple flow-limiting strategies, and determines the corresponding flow-limiting strategy to limit the flow based on the data type and communication protocol corresponding to the digital object, so as to meet the transmission requirements of the digital object of this data type. For example, sensor data (such as a temperature sensor only sends about 10 bytes of data per second) has extremely high requirements for real-time performance and a small amount of data. In view of the characteristics of frequent update and strong timeliness of such data, the corresponding sliding window flow-limiting algorithm is adopted to ensure the timely transmission of data. For the metadata of digital objects with strong timeliness, the generation time and timeliness requirements of the data are recorded. Based on these two, the timeliness situation of the digital object is determined. The generation time records the time point when the data is generated, which is extremely critical for data with timeliness requirements. The timeliness requirements will clarify the expiration date of the data or the specified transmission completion time. The metadata of the digital object also records the data size.

[0024] Specifically: when the client requests a corresponding digital object from the server, it sends request data to the server based on the requested digital object. The server parses the protocol header of the received request data to determine the communication protocol based on which the request data is sent from the client to the server. This communication protocol is the target communication protocol for the data transmission of this request data. Subsequently, the digital object requested based on this request data will be transmitted back to the client based on this target communication protocol. Among them, the target communication protocol based on which the client sends the request data to the server can at least include: HTTP protocol, gRPC protocol, and WebSocket protocol.

[0025] Step S2: Determine the target data type and priority value of the digital object according to the metadata in the requested digital object.

[0026] In this embodiment, the server determines the digital object requested by the received request data, and then extracts the data type of the digital object and the recorded priority value of the digital object from the metadata of the digital object. This data type is the target data type of the digital object. Among them, the data types of digital objects include but are not limited to text data, sensor data, images and video data, structured data, etc.

[0027] Step S3: Determine the corresponding first flow-limiting strategy according to the target communication protocol and the target data type.

[0028] In this embodiment, based on the target communication protocol on which the determined request data is sent to the server and the target data type of the digital object requested by the request data, through a pre-constructed mapping relationship, a flow-limiting policy corresponding to the target communication protocol and the target data type is determined, and this flow-limiting policy is determined as the first flow-limiting policy to be adopted by the digital object. Among them, the pre-constructed mapping relationship is a mapping relationship that maps flow-limiting policies to communication protocols and data types. An optional implementation manner of this mapping relationship is: when the data type is sensor data or the transmission protocol is the HTTP protocol, the corresponding flow-limiting policy is the sliding window algorithm flow-limiting policy; when the data type is file data with a data volume exceeding a set value or the transmission protocol is one of the two protocols, the Grpc protocol and the WebSocket protocol, the corresponding flow-limiting policy is the token bucket algorithm flow-limiting policy; when the data type is background periodic task data, the corresponding flow-limiting policy is the leaky bucket algorithm flow-limiting policy. It should be understood that this mapping relationship can also record the mapping relationships between other flow-limiting policies and other communication protocols and other data types. Only some mapping relationships are listed here.

[0029] Step S4: According to the priority value, correct the parameters of the first flow-limiting policy to obtain a second flow-limiting policy.

[0030] In this embodiment, after determining the first flow-limiting policy corresponding to the digital object through step S3, based on the priority value of the digital object, the relevant parameters of the first flow-limiting policy are corrected to obtain the corrected first flow-limiting policy, and this corrected first flow-limiting policy is the second flow-limiting policy to be adopted corresponding to the digital object. This parameter correction process is to correct the relevant parameters in the first flow-limiting policy adopted by the digital object with a higher priority value, so that under the second flow-limiting policy obtained after the correction of the relevant parameters, compared with the first flow-limiting policy before the correction, the flow-limiting degree for this digital object will be reduced. At the same time, the relevant parameters in the first flow-limiting policy adopted by the digital object with a lower priority value are corrected, so that under the second flow-limiting policy obtained after the correction of the relevant parameters, compared with the first flow-limiting policy before the correction, the flow-limiting degree for this digital object will be increased.

[0031] Step S5: Perform flow-limited transmission on the digital object through the second flow-limiting policy.

[0032] In this embodiment, after obtaining the second rate-limiting policies corresponding to the respective digital objects requested by each client through step S4, the respective digital objects are rate-limited and transmitted to the corresponding requesting clients through the respective corresponding second rate-limiting policies. By way of example, the digital objects requested by client A are a1 and a2, the second rate-limiting policy corresponding to digital object a1 is x1, and the second rate-limiting policy corresponding to digital object a2 is x2; the digital object requested by client B is b1, and the second rate-limiting policy corresponding to digital object b1 is y1; digital object a1 is rate-limited and transmitted to client A through the second rate-limiting policy x1, digital object a2 is rate-limited and transmitted to client A through the second rate-limiting policy x2, and digital object b1 is rate-limited and transmitted to client B through the second rate-limiting policy y1.

[0033] In this embodiment, in the case where the digital object belongs to data composed of multiple data types, such as in the case of a digital object composed of video, attached text, and real-time bullet screens, the second rate-limiting policy corresponding to the digital object will include multiple types of second rate-limiting policies, and the multiple types of second rate-limiting policies correspond one-to-one to the multiple data types included in the digital object. For example, in the case where the digital object includes sensor data and file data with a data volume exceeding a set value, for the file data with a data volume exceeding the set value in the digital object, the corresponding second rate-limiting policy is a sliding window algorithm rate-limiting policy, and for the sensor data in the digital object, the corresponding second rate-limiting policy is a token bucket algorithm rate-limiting policy. During the transmission process of returning the digital object to the client, the corresponding second rate-limiting policy is called based on the data type currently being transmitted in the digital object for rate-limited transmission. For example, if the currently transmitted data type is sensor data, it is rate-limited and transmitted through the token bucket algorithm rate-limiting policy.

[0034] A composite flow-limiting method for large-volume data transmission based on the digital networking. First, parse the protocol header of the request data to determine the target communication protocol for data transmission; determine the target data type and priority value of the digital object according to the metadata in the requested digital object; determine the corresponding first flow-limiting strategy according to the target communication protocol and the target data type; modify the parameters of the first flow-limiting strategy according to the priority value to obtain a second flow-limiting strategy; and perform flow-limiting transmission on the digital object through the second flow-limiting strategy. Thus, by combining multiple flow-limiting strategies, based on the data type recorded in the metadata of the digital object for data transmission and the communication protocol for data transmission, select a flow-limiting strategy that adapts to the current data type and communication protocol from multiple flow-limiting strategies to perform flow-limiting processing on the digital object, so as to meet the data transmission requirements in different scenarios, thereby improving the stability and reliability of the system. At the same time, based on the priority level of the digital object, adjust the corresponding flow-limiting strategy, so as to give a smaller flow-limiting degree to the digital object with a higher priority and a larger flow-limiting degree to the digital object with a lower priority, thereby optimizing the resource utilization in the data transmission process. Through the solution of this application, the risk of system overload caused by a large number of requests can be effectively prevented, the stability and availability of the core service can be guaranteed through an intelligent traffic control mechanism, and the QoS index can still be maintained under the impact of sudden traffic, realizing the dynamic balance between transmission efficiency and system reliability, and significantly improving the user service experience in high-load scenarios.

[0035] Combined with the above embodiments, in one implementation, the embodiments of the present application also provide a composite flow-limiting method for large-volume data transmission based on the digital networking. In this composite flow-limiting method for large-volume data transmission based on the digital networking, step S4 may include steps S41 to S42: Step S41: Determine the target parameter for parameter modification according to the first flow-limiting strategy.

[0036] In this embodiment, the types of parameters modified by the present application for different types of first flow-limiting strategies are different. Specifically: when the first flow-limiting strategy corresponding to the digital object is a sliding window algorithm flow-limiting strategy, the target parameter for parameter modification is determined to be the window size. When the first flow-limiting strategy corresponding to the digital object is a token bucket algorithm flow-limiting strategy, the target parameter for parameter modification is determined to be the token generation rate. When the first flow-limiting strategy corresponding to the digital object is a leaky bucket algorithm flow-limiting strategy, the target parameters for parameter modification are determined to be the leaky bucket capacity and the water leakage rate.

[0037] Step S42: Modify the target parameter according to the priority value and a preset modification rule to obtain a second flow-limiting strategy.

[0038] In this embodiment, the present application pre - establishes a preset correction rule. Based on the target parameter for parameter correction corresponding to the digital object and the priority value of the digital object, a correction amount corresponding to both the target parameter and the priority value is determined from the preset correction rule. The target parameter in the first current - limiting policy of the digital object is corrected by the correction amount to obtain the second current - limiting policy corresponding to the digital object. Subsequently, the digital object is subjected to current - limiting transmission through the second current - limiting policy.

[0039] Combined with the above - mentioned embodiments, in one implementation manner, the embodiment of the present application further provides a composite current - limiting method for large - flow data transmission based on the digital networking. In this composite current - limiting method for large - flow data transmission based on the digital networking, step S42 may include steps S421 to S423: Step S421: Determine the basic priority value according to the priority values of all digital objects in the cache queue.

[0040] In this embodiment, multiple digital objects to be returned to the request client are cached in the cache queue. Based on the priority values of all digital objects in the current cache queue, a basic priority value is determined therefrom. One optional implementation manner is: sort the priority values of all digital objects in the current cache queue according to the value size, and take the priority value corresponding to the median as the basic priority value. For example, if the result of sorting the priority values of all digital objects in the current cache queue according to the value size is 1, 3, 3, 5, 7, the middle 3 is the median, so the basic priority value is determined to be 3; if the result of sorting the priority values of all digital objects in the current cache queue according to the value size is 1, 2, 3, 3, 5, 6, there are two middle values, which are 3 and 3 respectively, so the average of the two is taken as the median, that is, the basic priority value is determined to be 3.

[0041] Step S422: Determine the priority deviation value between the priority value of the digital object and the basic priority value.

[0042] In this embodiment, after determining the basic priority value through step S421 based on the priority values of all digital objects to be transmitted in the cache queue. The priority value of the digital object to be transmitted in the current cache queue is subtracted from the basic priority value to obtain the priority deviation value between the priority value of the digital object and the basic priority value. Each digital object to be transmitted in the cache queue will obtain a priority deviation value corresponding to itself through the same subtraction implementation manner as above.

[0043] Step S423: Based on the priority deviation value, the base value and adjustment step of the target parameter, and the target percentage value, perform parameter correction on the target parameter to obtain a second flow-limiting policy, where the target percentage value is a value corresponding to both the priority value of the digital object and the type of the target parameter.

[0044] In this embodiment, based on the priority value of the digital object to be transmitted in the current cache queue and the type of the target parameter, determine the target percentage value corresponding to both the priority value and the type of the target parameter. That is, when determining the target percentage value of a digital object, based on the priority value of the digital object and the type of the target parameter corresponding to the digital object, determine a percentage value that matches both of them as the target percentage value of the digital object. At the same time, this application sets corresponding base values and adjustment steps for different types of target parameters. After obtaining the priority deviation value and the target percentage value corresponding to the digital object in the current cache queue through steps S421 to S423, perform parameter correction on the target parameter of the digital object through a preset algorithm using the priority deviation value, the target percentage value, and the preset base value and adjustment step of this type of target parameter to obtain the second flow-limiting policy of the digital object. The difference between the second flow-limiting policy obtained through parameter correction and the first flow-limiting policy before parameter correction is that the target parameter in the first flow-limiting policy has changed, from the initial base value, and has been corrected through the calculation of the determined relevant parameters and the preset algorithm. The expression of the preset algorithm is: y = x + (a - m) × w × L, where y represents the value after parameter correction of the target parameter, x is the base value of the target parameter, a is the priority value of the digital object, m is the priority deviation value, w is the target percentage value corresponding to both the priority value of the digital object and the type of the target parameter, and L is the adjustment step of the target parameter.

[0045] Combined with the above embodiments, in one implementation, the embodiments of this application also provide a composite flow-limiting method for large-flow data transmission based on the digital networking. In this composite flow-limiting method for large-flow data transmission based on the digital networking, before step S4, the method further includes step S04: Adjust the priority value of the digital object according to the waiting duration of the digital object in the cache queue.

[0046] In this embodiment, new digital objects are continuously stored in the cache queue, and there may be a situation where some digital objects stay in the cache queue for a long time. If a new digital object with a higher priority than such a digital object that stays in the cache queue for a long time is stored in the cache queue subsequently, and if the same lower flow-limiting degree is given to this new digital object, and because such a digital object that stays in the cache queue for a long time has a lower priority than this new digital object and a higher flow-limiting degree is still given to such a digital object that stays in the cache queue for a long time, this will further exacerbate the residence time of such a digital object that stays in the cache queue in the cache queue, resulting in unreasonable resource allocation for data transmission in the entire system, that is, some low-priority digital objects in the cache queue have not been allocated appropriate resources for data transmission for a long time. Therefore, this application will dynamically adjust the priority value of the digital object based on the waiting time of the digital object in the cache queue to avoid the occurrence of the above situation, thereby ensuring more reasonable resource allocation for data transmission in the entire system. An optional implementation manner is: multiply the waiting time of the digital object in the cache queue by a preset percentage to obtain the priority correction amount corresponding to the digital object, and then add the priority correction amount to the priority value recorded in the metadata of the digital object to obtain the adjusted priority value corresponding to the digital object. The preset percentage can be set according to the actual application scenario and will not be specifically limited here. For example, the priority of digital object a in the cache queue is 3, the waiting time of digital object a in the cache queue is 200 ms, and the preset percentage preset in advance is 0.1%. Accordingly, the priority correction amount corresponding to digital object a is 200×0.1% = 2, and the adjusted priority value of digital object a is 5.

[0047] In this embodiment, the method of dynamically adjusting the priority value of the digital object in real time based on the waiting time of the digital object in the cache queue has a too high adjustment frequency, which is likely to cause waste of system resources. At the same time, the entire flow-limiting system will frequently re-evaluate and process the digital objects in the cache queue based on the adjusted priority, resulting in frequent system responses and reducing the processing efficiency of the system. To solve this problem, for the adjustment of the priority value of the digital object according to the waiting time of the digital object in the cache queue in this application, another optional implementation manner is provided to reduce the adjustment frequency of the priority value and reduce the system response time.

[0048] In this embodiment, another optional implementation manner is as follows: Determine whether the waiting duration of the digital object in the cache queue is greater than or equal to a duration threshold, which can be set according to the actual application scenario and is not specifically limited herein. For example, it can be taken as 100 ms, 200 ms, etc. When the waiting duration of the digital object in the cache queue is not greater than or equal to the duration threshold, keep the priority value of the digital object unchanged and do not adjust the priority value of the digital object. When the waiting duration of the digital object in the cache queue is greater than or equal to the duration threshold, start dynamically adjusting the priority value of the digital object. The adjustment strategy is that for every increase in the set duration of the waiting duration of the digital object in the cache queue, increase the priority value recorded in the metadata of the digital object by a set percentage. Among them, the set duration can be set according to the actual application scenario and is not specifically limited herein. For example, it can be taken as 50 ms, 100 ms, etc. The set percentage can be set according to the actual application scenario and is not specifically limited herein. For example, it can be taken as 10%, 20%, etc. It should be understood that the priority value of the digital object has an upper limit, and the priority of the digital object will not be increased any further after it is adjusted to the highest priority during the dynamic adjustment of the priority value of the digital object.

[0049] Exemplarily, the preset duration threshold is 200 ms, the preset set duration is 50 ms, and the preset set percentage is 10%. The waiting durations of digital objects a and b in the cache queue are 150 ms and 200 ms respectively, and the priority values recorded in their respective metadata are 3 and 5. The waiting duration of digital object a in the cache queue is less than the set duration, so the priority value of digital object a remains unchanged. The waiting duration of digital object b in the cache queue is equal to the set duration, so the priority value of this digital object is dynamically adjusted. Since the waiting duration of digital object b in the cache queue is 200 ms and the preset set duration is 50 ms, the waiting duration of this digital object b in the cache queue has increased by 4 set durations. And for each set duration, the set percentile increased on the basis of the priority value recorded in the metadata of this digital object b is 10%. Accordingly, the current priority value of this digital object b will be dynamically adjusted to 5×(1 + 40%) = 7. Accordingly, the priority value of this digital object b will be dynamically adjusted to 7. In this application, the larger the priority value, the smaller the corresponding current limiting degree. When the waiting duration of digital object b in the cache queue continues to become 240 ms, since the set duration has not increased, the priority value will not be adjusted and remains at 7 after the previous adjustment. When the waiting duration of digital object b in the cache queue continues to become 250 ms, since another set duration has increased, the priority value will continue to be adjusted and will be dynamically adjusted to 5×(1 + 50%) = 7.5.

[0050] In this application, when step S04 is further included before step S4 of a composite current-limiting method for large-flow data transmission based on the digital object network, step S4 may include: correcting the parameters of the first current-limiting policy according to the adjusted priority value to obtain a second current-limiting policy.

[0051] In this embodiment, when step S04 is further included before step S4 of a composite current-limiting method for large-flow data transmission based on the digital object network, the priority value of the digital object used in the process of correcting the parameters of the corresponding first current-limiting policy according to the priority value of the digital object to obtain the corresponding second current-limiting policy in step S4 is the priority value of the digital object after each dynamic adjustment.

[0052] Specifically: when the digital object is initially stored in the cache queue, the waiting time of the digital object in the cache queue is not greater than or equal to the time threshold. Accordingly, the parameters of the first current-limiting policy corresponding to the digital object are corrected based on the priority value recorded in the metadata of the digital object to obtain a second current-limiting policy, and then the digital object is transmitted with current limiting according to the second current-limiting policy. When the digital object has not been transmitted with current limiting for a long time or has not been completely transmitted to the requesting client, and at the same time the waiting time in the cache queue is greater than or equal to the time threshold, the priority value of the digital object will be dynamically adjusted at this time. The adjustment method is the same as the above embodiment. Then, the parameters of the first current-limiting policy corresponding to the digital object are corrected according to the adjusted priority value of the digital object to obtain a new second current-limiting policy, and then the subsequent data of the digital object is transmitted with current limiting according to the new second current-limiting policy.

[0053] Combined with the above embodiments, in one implementation, the embodiments of this application further provide a composite current-limiting method for large-flow data transmission based on the digital object network. In this composite current-limiting method for large-flow data transmission based on the digital object network, before step S5, the method further includes steps S051 to S052: Step S051: Determine the actual measured value of the network traffic during the data transmission process.

[0054] In this embodiment, when the present application determines the second rate-limiting strategy of a digital object based on the data type, priority value, and communication protocol of the data transmission of the digital object, only the attribute data of the digital object itself is used to optimize and adjust the rate-limiting strategy of the digital object to meet the transmission requirements of the digital object. However, the network traffic situation of the entire system is not considered in this process, which may lead to unreasonable allocation of the network traffic of the entire system, resulting in overload or excessive traffic surplus. For example, during the entire process of determining the second rate-limiting strategy of the digital object, only the data type of the digital object is considered to meet the transmission requirements of the digital object. At the same time, the priority of the digital object is considered, and the rate-limiting strategy is appropriately adjusted to give a smaller rate-limiting degree to the digital object with a higher priority and a larger rate-limiting degree to the digital object with a lower priority. When each digital object performs rate-limiting transmission using its corresponding second rate-limiting strategy, the actual network traffic situation of the entire system is not taken into account, which may result in data transmission overload or excessive traffic surplus in the entire system. To solve this problem, before step S5, the present application further adjusts the second rate-limiting strategy corresponding to each digital object in the cache queue based on the relationship between the actual measured value of the network traffic of the entire system during the data transmission process and the set value of the network traffic of the entire system. The set value of the network traffic is the desired network traffic value when the entire system is in a suitable state. First, the actual measured value of the network traffic during the current data transmission process is determined.

[0055] Step S052: According to the actual measured value and the set value of the network traffic, dynamically adjust the second rate-limiting strategy through PID feedback adjustment.

[0056] In this embodiment, based on the determined actual measured value of the network traffic during the current data transmission process and the pre-set set value of the network traffic, the second rate-limiting strategy corresponding to each digital object in the cache queue is dynamically adjusted through PID feedback adjustment.

[0057] In the present application, when steps S051 to S052 are further included before step S5, step S5 may include: performing rate-limiting transmission on the digital object through the dynamically adjusted second rate-limiting strategy.

[0058] In this embodiment, when this application includes dynamically adjusting the second rate-limiting policy corresponding to each digital object in the cache queue through steps S051 to S052, the implementation manner of step S5 is as follows: For each digital object in the cache queue, perform rate-limiting transmission on itself based on the dynamically adjusted second rate-limiting policy corresponding to itself. For example, the cache queue includes a digital object a1, and the dynamically adjusted second rate-limiting policy corresponding to this digital object a1 is z1; and includes a digital object a2, and the dynamically adjusted second rate-limiting policy corresponding to this digital object a2 is z2; and includes a digital object a3, and the dynamically adjusted second rate-limiting policy corresponding to this digital object a3 is z3. Accordingly, perform rate-limiting transmission on the digital object a1 through z1, perform rate-limiting transmission on the digital object a2 through z2, and perform rate-limiting transmission on the digital object a3 through z3.

[0059] Combined with the above embodiments, in one implementation manner, the embodiment of this application also provides a composite rate-limiting method for large-flow data transmission based on the digital networking. In this composite rate-limiting method for large-flow data transmission based on the digital networking, step S052 may include: When the second rate-limiting policy is the token bucket algorithm, dynamically adjust the token bucket capacity or the token generation rate in the second rate-limiting policy through PID feedback regulation according to the actual measurement value and the set value of the network traffic; when the second rate-limiting policy is the leaky bucket algorithm, dynamically adjust the leaky bucket capacity or the water outlet rate in the second rate-limiting policy through PID feedback regulation according to the actual measurement value and the set value of the network traffic; when the second rate-limiting policy is the sliding window algorithm, dynamically adjust the window size or the request processing rate in the second rate-limiting policy through PID feedback regulation according to the actual measurement value and the set value of the network traffic.

[0060] In this embodiment, the control process of PID feedback regulation includes proportional control (P), integral control (I), and derivative control (D). Proportional control is used to adjust the data transmission rate or bandwidth allocation according to the deviation between the set value of network traffic and the actual measured value of network traffic, that is, to adjust the output of the controller according to the magnitude of the current network traffic error. This control determines the response intensity of the controller to the current deviation. Integral control is used to adjust the data transmission rate or bandwidth allocation according to the cumulative deviation between the set value of network traffic and the actual measured value of network traffic, that is, to adjust the output of the controller according to the cumulative amount of the current network traffic error over time. This control can eliminate the steady-state error of the system, ensure that the system finally stabilizes near the set value, and avoid deviation. Derivative control is used to adjust the data transmission rate or bandwidth allocation according to the change rate between the set value of network traffic and the actual measured value of network traffic, that is, to adjust the output of the controller according to the change rate of the current network traffic error. This control can predict the future change trend of network traffic, adjust the transmission rate in advance, reduce congestion and latency, and improve network performance. This PID feedback regulation adjusts the relevant parameters in the second rate-limiting policy for each digital object in the cache queue based on the relationship between the set value of network traffic and the actual measured value of network traffic in each control item, so as to achieve the purpose of adjusting the data transmission rate or bandwidth allocation, and thus make the actual measured value of network traffic as stable as possible near the set value. When the second rate-limiting policy is the token bucket algorithm, according to the relationship between the actual measured value of network traffic and the set value of network traffic, when the PID feedback regulation adjusts the token bucket algorithm, what is adjusted is the token bucket capacity or token generation rate of the token bucket algorithm. When the second rate-limiting policy is the leaky bucket algorithm, according to the relationship between the actual measured value of network traffic and the set value of network traffic, when the PID feedback regulation adjusts the leaky bucket algorithm, what is adjusted is the leaky bucket capacity or water output rate of the leaky bucket algorithm. When the second rate-limiting policy is the sliding window algorithm, according to the relationship between the actual measured value of network traffic and the set value of network traffic, when the PID feedback regulation adjusts the sliding window algorithm, what is adjusted is the window size or request processing rate of the sliding window algorithm.

[0061] In this embodiment, as Figure 2As shown in the figure, a composite current-limiting method for large-traffic data transmission based on the digital networking is provided in this application. First, the request data initiated by the client is received and security rule detection is performed. When it is determined that the request data is qualified, the digital object of the request is determined, and this digital object is used as the digital object to be transmitted and waits to be returned to the client. Metadata parsing is performed on the digital object to be transmitted to determine the target data type and priority value of this digital object. At the same time, the communication protocol of the request data corresponding to this digital object is identified to determine the target communication protocol to be adopted. Based on the target data type and target communication protocol corresponding to this digital object, a current-limiting strategy matching the target data type and target communication protocol is determined, and this current-limiting strategy is determined as the first current-limiting strategy of this digital object. Then, further parameter correction is performed on the target parameters of the first current-limiting strategy of this digital object based on the priority value of this digital object to obtain the second current-limiting strategy of this digital object. Next, based on the relationship between the actual measured value of the current network traffic and the set value of the network traffic, the corresponding parameters of the second current-limiting strategy of this digital object are dynamically adjusted through PID feedback control, so that the actual measured value of the network traffic during the digital object transmission process of the entire system is stabilized near the set value. Finally, the digital object is subjected to current-limiting transmission through the second current-limiting strategy of this digital object after PID feedback control.

[0062] In this embodiment, the current-limiting process of the token bucket algorithm is as follows: As Figure 3 shown, a token bucket is set, and there is a set number of tokens in the initial state of this token bucket. The token generation rate is a parameter that can be dynamically adjusted. When a digital object is transmitted using this token bucket algorithm, it is checked whether there are enough tokens in the token bucket. If there are enough tokens, the corresponding number of tokens is taken out of the token bucket, and the digital object is allowed to perform data transmission; if the tokens are insufficient, the data transmission of this digital object is temporarily shelved. During the current-limiting process, new tokens will be continuously generated and put into the token bucket to ensure that it can respond in a timely manner when there is a burst traffic demand.

[0063] In this embodiment, the current-limiting process of the leaky bucket algorithm is as follows: As Figure 4 shown, the leaky bucket drains the digital object data from the leaky bucket at a fixed rate, and this drain rate is the average rate of data transmission. When there is digital object data that needs to be transmitted, the digital object data first enters the leaky bucket. No matter how fast the digital object data flows in, the leaky bucket always transmits the digital object data at a stable rate. This can ensure that the average rate of digital object data transmission does not exceed the preset limit and guarantee long-term stable transmission. If the digital object data flows in too fast during a certain period of time, the leaky bucket will temporarily store the excess digital object data and wait for enough time to transmit it at a stable rate.

[0064] In this embodiment, the current-limiting process of the sliding window algorithm is as follows: AsFigure 5 As shown, set the window size, that is, the amount of data allowed to be sent within each set time range, and set the maximum amount of data allowed to be sent per unit time. Divide the set time range into multiple time segments, and record the transmission volume of digital object data within each time segment. During the whole process, check whether the data transmission volume within the current window exceeds the allowed data transmission volume. If it does not exceed, continue to allow the transmission of digital object data; if it exceeds, temporarily reject the transmission of the digital object data and wait for the next window to try again. The system continuously monitors the movement of the window to ensure the accuracy and real-time nature of flow limiting.

[0065] Combined with the above embodiments, in one implementation, the embodiments of the present application further provide a composite flow limiting method for large flow data transmission based on the digital networking. In this composite flow limiting method for large flow data transmission based on the digital networking, the method further includes: when the priority values of all digital objects in the cache queue are adjusted to the highest priority, perform average resource allocation for the data transmission of each digital object.

[0066] In this embodiment, after the priority values of all digital objects in the cache queue reach the highest priority, based on the set value of the network traffic, perform average resource allocation for the data transmission of each digital object in the cache queue.

[0067] Combined with the above embodiments, in one implementation, the embodiments of the present application further provide a composite flow limiting method for large flow data transmission based on the digital networking. In this composite flow limiting method for large flow data transmission based on the digital networking, when the target communication protocol is the HTTP protocol, the method further includes: filter the request data through a regular rule library to determine whether the request data belongs to a known attack pattern; in the case of not hitting the rule, analyze the parameter semantics of the request data through an AST parsing engine to determine whether the request data is a suspicious request; in the case of the request data being a suspicious request, direct the request data to an isolation environment for execution and perform abnormal behavior monitoring; in the case of not detecting abnormal behavior, respond to the request data.

[0068] In this embodiment, flow limiting during the data transmission process is not only a means of performance optimization but also an important measure for security protection. The present application will combine the security rules of the communication protocol to more effectively defend against various security threats.

[0069] Specifically: when the communication protocol for data transmission is the HTTP protocol, the client initiates a request and encapsulates the request into request data in the form of HTTP and sends it to the server. After the server receives the request data, it uses the regular expression rule set of OWASP (Open Web Application Security Project) (CRS, which contains more than 3,000 signature rules for various attacks) to check whether the request data contains specific XSS patterns, such as <script>.< / script> , and other known SQL injection or XSS attack patterns. If the request data matches a signature in the regular expression rule library, the system will mark the request data as a potential attack, reject it, or perform further in-depth inspection on the request data. In the case where the request data does not match the signature in the regular expression rule library, the system will use a parser based on AST (Abstract Syntax Tree) to perform in-depth lexical and syntactic analysis on the parameters in the request data to identify more complex non-standard character combinations or code structures that may constitute SQL injection or XSS attacks. After identifying potential attack code, the system will further mark the request data as a suspicious request and prepare for the next step of dynamic sandbox detection. For the request data marked as a suspicious request, the system redirects the request data to an isolated execution environment (i.e., sandbox) for detection. In the sandbox, the system will simulate the execution of the code segment in the request data while closely monitoring any abnormal database operations or DOM tampering behaviors. If abnormal behaviors are detected, the system will confirm that the request data is an attack request and immediately block the further execution of the request data. If no abnormal behaviors are detected, it is confirmed that the request data is secure and processed normally.

[0070] Combined with the above embodiments, in one implementation, the embodiment of the present application further provides a composite current limiting method for large traffic data transmission based on the data Internet of Things. In the composite current limiting method for large traffic data transmission based on the data Internet of Things, when the target communication protocol is the gRPC protocol, the method further includes: verifying the RPC method signature in the request data to determine whether the request data is legal, where the RPC method signature is a unique signature generated by the server for each RPC method; when the request data is legal, determining whether the call chain of the request data is an abnormal call chain according to the request data and the service dependency graph, where the service dependency graph is a dependency relationship between services constructed based on the information recorded in each data request; or determining whether the request data is legal according to the dynamic token in the request data, where the dynamic token is sent to the client after the server passes the verification of the certificate and fingerprint of the connection request of the client sending the request data; when the request data is legal, responding to the request data.

[0071] In this embodiment, when the communication protocol for data transmission is the HTTP protocol, the client, according to the business logic, prepares the RPC method to be called, the RPC method signature corresponding to the RPC method, and the parameters required for the RPC method, and encapsulates them into request data. The RPC method signature is a unique signature (such as SHA256) generated by the server for each RPC method and is used to sign the request data. After receiving the request data, the server first checks whether the RPC method signature in the request data matches the signature pre-stored by the server to ensure that the request data has not been tampered with and is indeed a call to the expected RPC method. When the request data passes the signature verification, the server analyzes whether the call chain of the request data is legal based on the request data and the service dependency graph. The service dependency graph is constructed by the server based on a large number of historical request data records and describes the dependency relationships between services. For example, a normal call chain may be "user service → order service → payment service", while a reverse call such as "payment service → log service" may be abnormal. The server checks whether the call chain represented by the current request data conforms to the expected service dependency relationship. When the call chain of the request data is abnormal, the server immediately blocks the request data to prevent potential security risks.

[0072] In this embodiment, another alternative implementation is as follows: The client establishes a connection with the server through two-way TLS (Transport Layer Security) encryption. During this process, the client and the server perform two-way authentication when establishing the connection. The client presents its client certificate to the server, and the server also presents its server certificate to the client. After receiving the client certificate, the server verifies the validity of the client certificate, whether it is issued by a trusted certificate authority, and whether the information in the certificate (such as the client's public key, fingerprint, etc.) matches the expectation. At the same time, the server may check other attributes in the client certificate, such as the validity period of the certificate, the client's identity identifier, etc., to ensure that the client has the permission to access the required RPC service. After successfully verifying the legitimacy of the client, the server generates a dynamic token and sends the dynamic token to the client. The dynamic token is an encrypted string that contains information such as the client's identity information, the generation time of the token, and the validity period. The dynamic token has timeliness and is valid for a certain period of time after being generated. After the validity period expires, the client needs to request a new dynamic token from the server again. After receiving the dynamic token, the client saves the dynamic token locally and records the dynamic token in subsequent RPC requests. When the client calls a certain RPC method, the name and parameters of the called RPC method and the dynamic token are encapsulated into request data and sent to the server.

[0073] In this embodiment, after the server receives the request data from the client, it verifies whether the dynamic token in the request data has expired, whether it has been tampered with, and whether the identity information in the dynamic token matches the client that initiated the request. If the dynamic token verification passes, it determines that the request data is legal and executes the corresponding RPC method according to the legal request data. After the execution is completed, the server generates response data.

[0074] In this embodiment, another alternative implementation is as follows: Based on the call frequency, parameter size, and response error rate of the system's RPC methods, it is determined whether the current is in an abnormal request process. If it is determined that the current is in an abnormal request process, corresponding security measures are taken, such as logging, sending alerts, or activating a circuit breaker mechanism. Specifically: Analyze the number of requests received by the system per unit time to determine whether there are abnormal high-frequency requests in the system, and monitor the size of the request parameters to determine possible parameter injection attacks or resource exhaustion attacks. For example, continuously transmit data streams larger than 10KB, and analyze the error rate of the system response to determine whether the system is under attack or about to crash. When one of the call frequency, parameter size, and response error rate exceeds its corresponding normal range, an abnormal determination is triggered, and at this time, corresponding security measures are taken, such as logging, sending alerts, or activating a circuit breaker mechanism.

[0075] Combined with the above embodiments, in one implementation, the embodiments of the present application further provide a composite flow limiting method for large-flow data transmission based on the data networking. In this composite flow limiting method for large-flow data transmission based on the data networking, when the target communication protocol is the WebSocket protocol, the method further includes: performing binary magic number verification on the request data to determine whether the request data is legal, and / or, performing malicious keyword scanning on the request data to determine whether the request data is legal; when the request data is legal, responding to the request data; adjusting the connection number of each client according to the CPU usage rate of the server; disconnecting the request connections that meet the preset heartbeat monitoring mechanism according to the preset heartbeat monitoring mechanism.

[0076] In this embodiment, the client first constructs request data in the form of WebSocket and sends the constructed request data to the server. After receiving the request data, the server first performs binary magic number verification, specifically by checking whether the beginning of the request data contains a specific binary sequence (magic number). The magic number is part of the WebSocket protocol and is used to distinguish WebSocket requests from other types of network requests. If the beginning of the request data does not meet the magic number requirements of the WebSocket protocol, the server will consider the request illegal and will reject the connection or return an error response. For request data in text format, the server uses the Aho-Corasick automaton algorithm to perform real-time malicious keyword scanning. If any malicious keyword is included in the request data, it is determined that the request data is illegal, and the connection will be rejected, the existing connection will be disconnected, or other security measures will be taken. After establishing a connection with the client, the server dynamically adjusts the maximum connection number for each client according to the current CPU usage rate. For example, when the CPU usage rate reaches 70%, the server may reduce the maximum connection number per IP from 100 to 30 to reduce the server load and avoid performance issues. Once the WebSocket connection is established, heartbeat frames (such as PING frames) will be regularly sent between the server and the client to maintain the connection. The server sets up a heartbeat detection mechanism, requiring the client to send a PING frame every 30 seconds. If the client fails to respond twice in a row (i.e., does not send a PING frame or does not correctly reply with a PONG frame), the server will consider the connection to have expired and will disconnect the connection. In addition, the server also performs a countdown on idle connections. If a connection has no activity (i.e., does not send or receive any data) within a preset duration (such as within 30 minutes), the server will automatically release the resources occupied by the connection. After the WebSocket connection is established and successfully passes the heartbeat detection, two-way communication can be carried out between the client and the server. The client can send data to the server, and the server can also send data to the client. During the communication process, the server will continue to monitor the CPU usage rate and the connection status, and adjust the connection number or disconnect non-compliant connections as needed.

[0077] Through a composite current-limiting method for large-flow data transmission based on the digital networking provided by this application, not only can the system more effectively defend against security threats such as SQL injection, XSS attacks, and RPC abuse through specific security rules of the communication protocol, thereby improving the overall security. It also enables the system to automatically select the most suitable current-limiting strategy according to the metadata of digital objects through the application of multi-level current limiting and multi-strategy combination, optimizing traffic management and improving the stability and response speed of the system. In addition, the coherent data transmission and automated current-limiting strategy ensure a smooth experience for users when sending different types of data. Whether it is a video or a text file, it will not be interrupted due to current limiting, enhancing user satisfaction. At the same time, through metadata semantic parsing and the generation of dynamic policy chains, the deep coupling of current-limiting rules and data transmission logic is achieved. For example, when transmitting a digital object composed of a video, attached text, and real-time bullet screens, the system will automatically allocate independent current-limiting strategies according to the metadata of each part while maintaining the coherence of the overall transmission session.

[0078] In the overall architecture of the system of this application, the digital object identification system creates a unique identification ID to ensure the identifiability of digital objects in the system and closely cooperates with the digital object registration system. The registration system extracts metadata to provide a decision basis for current limiting. The digital object warehouse system is responsible for efficiently storing digital objects for quick retrieval and access. As the core, current limiting flexibly selects current-limiting methods based on metadata, such as combining the advantages of the sliding window, token bucket, and leaky bucket algorithms, and dynamically switching or combining them according to different data scenarios. For example, when the network load is light and the priority of the transmitted data is not high, the sliding window current-limiting threshold can be appropriately relaxed or the token generation rate of the token bucket algorithm can be reduced to improve transmission efficiency; when high-priority data appears, quickly adjust the strategy to ensure its bandwidth. This architecture effectively supports the implementation of the composite current-limiting method through the collaborative work of each component, ensuring that the system can flexibly respond to different situations and guarantee the stability and efficiency of data transmission.

[0079] In the Internet of Things (IoT) domain under the Digital Internet of Things (DIoT), a large number of sensors continuously generate data. For example, environmental monitoring sensors send a small amount of data per second with extremely high real-time requirements. This application ensures the timely transmission of data by means of sliding window rate limiting according to the characteristics of sensor data. Meanwhile, a dynamic feedback mechanism is utilized to adjust the sliding window parameters (such as shortening the window time or increasing the request threshold) in a timely manner when the network condition changes (such as an increase in latency), thereby guaranteeing the real-time nature of data and the stability of the system. For the data transmission of production equipment and the issuance of control instructions in the Industrial Internet of Things under the DIoT, the data volume of equipment transmission is large and there may be burst traffic (such as a large amount of status data being uploaded when a device suddenly fails). At the same time, the control instructions have high real-time requirements. The token bucket algorithm of this application can effectively support burst traffic, ensure the timely upload of device status data, and guarantee the continuity and stability of the production process by preferentially ensuring the transmission bandwidth of high-priority control instructions (based on the priority information in the metadata). For example, in an automated production line, when an emergency failure occurs, high-priority fault alarm instructions can be transmitted preferentially, and at the same time, the rate limiting strategy of other low-priority data (such as device normal operation status monitoring data) is dynamically adjusted to ensure that critical instructions are quickly conveyed to the control center and measures are taken in a timely manner to avoid losses. In the big data analysis scenario under the DIoT, the data sources are diverse, including real-time data and batch data. For real-time data streams (such as real-time user behavior data for immediate analysis), sliding window rate limiting is adopted to ensure the freshness of data and the timeliness of analysis; for batch data transmission (such as historical data for model training), the token bucket algorithm is used to balance the transmission efficiency and system resource occupancy. Moreover, according to the timeliness requirements of data (such as the value of some real-time analysis data decreasing when it becomes outdated) and importance (such as core business data being processed preferentially), personalized rate limiting is carried out through metadata information to ensure the efficient and stable operation of the big data analysis system under different loads and data requirements.

[0080] Based on the same inventive concept, this application provides a composite rate limiting system for large-volume data transmission based on the Digital Internet of Things, as Figure 6 shown. The system 600 includes: A target communication protocol determination module 601, configured to parse the protocol header of the request data to determine the target communication protocol for data transmission; A metadata parsing module 602, configured to determine the target data type and priority value of the digital object according to the metadata in the requested digital object; A first rate limiting strategy determination module 603, configured to determine the corresponding first rate limiting strategy according to the target communication protocol and the target data type; A second rate limiting strategy determination module 604, configured to correct the parameters of the first rate limiting strategy according to the priority value to obtain a second rate limiting strategy; The flow limiting control module 605 is used to perform flow limiting transmission on the digital object through the second flow limiting policy.

[0081] Optionally, the second flow limiting policy determination module 604 includes: The target parameter determination module is used to determine the target parameter for parameter correction according to the first flow limiting policy; The parameter correction module is used to perform parameter correction on the target parameter according to the priority value and the preset correction rule to obtain the second flow limiting policy.

[0082] Optionally, the parameter correction module includes: The basic priority value determination module is used to determine the basic priority value according to the priority values of all digital objects in the cache queue; The priority deviation value determination module is used to determine the priority deviation value between the priority value of the digital object and the basic priority value; The parameter correction sub-module is used to perform parameter correction on the target parameter according to the priority deviation value, the basic value and the adjustment step size of the target parameter, and the target percentage value, to obtain the second flow limiting policy, where the target percentage value is a value corresponding to both the priority value of the digital object and the type of the target parameter.

[0083] Optionally, the system 600 further includes: The priority adjustment module is used to adjust the priority value of the digital object according to the waiting duration of the digital object in the cache queue; The second flow limiting policy determination module 604 is used to perform parameter correction on the first flow limiting policy according to the adjusted priority value to obtain the second flow limiting policy.

[0084] Optionally, the system 600 further includes: The actual measurement value determination module is used to determine the actual measurement value of the network traffic during the data transmission process; The dynamic adjustment module is used to perform dynamic adjustment on the second flow limiting policy through PID feedback adjustment according to the actual measurement value and the set value of the network traffic; The flow limiting control module 605 is used to perform flow limiting transmission on the digital object through the dynamically adjusted second flow limiting policy.

[0085] Optionally, the dynamic adjustment module includes: The first dynamic adjustment module is used to, when the second flow limiting policy is the token bucket algorithm, perform dynamic adjustment on the token bucket capacity or the token generation rate in the second flow limiting policy through PID feedback adjustment according to the actual measurement value and the set value of the network traffic; A second dynamic adjustment module, configured to, when the second flow limiting policy is the leaky bucket algorithm, dynamically adjust the leaky bucket capacity or the water output rate in the second flow limiting policy through PID feedback adjustment according to the actual measurement value and the set value of the network traffic; A third dynamic adjustment module, configured to, when the second flow limiting policy is the sliding window algorithm, dynamically adjust the window size or the request processing rate in the second flow limiting policy through PID feedback adjustment according to the actual measurement value and the set value of the network traffic.

[0086] Optionally, the system 600 further includes: A resource allocation module, configured to perform average resource allocation for data transmission of each digital object when the priority values of all digital objects in the cache queue are adjusted to the highest priority.

[0087] Optionally, the system 600 further includes: A first security module, configured to, when the target communication protocol is the HTTP protocol, filter the request data through a regular rule library to determine whether the request data belongs to a known attack pattern; A second security module, configured to, when the rule is not hit, analyze the parameter semantics of the request data through an AST parsing engine to determine whether the request data is a suspicious request; A third security module, configured to, when the request data is a suspicious request, direct the request data to an isolation environment for execution and perform abnormal behavior monitoring; A first response module, configured to respond to the request data when no abnormal behavior is detected.

[0088] Optionally, the system 600 further includes: A fourth security module, configured to, when the target communication protocol is the gRPC protocol, verify the RPC method signature in the request data to determine whether the request data is legal, where the RPC method signature is a unique signature generated by the server for each RPC method; A fifth security module, configured to, when the request data is legal, determine whether the call chain of the request data is an abnormal call chain according to the request data and the service dependency graph; A sixth security module, configured to block the abnormal call chain when the call chain is an abnormal call chain, where the service dependency graph is a dependency relationship between services constructed based on information recorded for each data request; The seventh security module is used to determine whether the request data is legal according to the dynamic token in the request data. The dynamic token is sent to the client after the server passes the certificate and fingerprint verification of the connection request of the client sending the request data. The second response module is used to respond to the request data when the request data is legal.

[0089] Optionally, the system 600 further includes: The eighth security module is used to perform binary magic number verification on the request data to determine whether the request data is legal when the target communication protocol is the WebSocket protocol, and / or perform malicious keyword scanning on the request data to determine whether the request data is legal; The third response module is used to respond to the request data when the request data is legal; The first connection adjustment module is used to adjust the connection number of each client according to the CPU usage rate of the server; The second connection adjustment module is used to disconnect the request connection that meets the preset heartbeat monitoring mechanism according to the preset heartbeat monitoring mechanism.

[0090] For the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, please refer to the partial description of the method embodiment.

[0091] It should be noted that for the method embodiment, for the sake of simple description, it is expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of the present application are not limited by the described action sequence, because according to the embodiments of the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0092] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other.

[0093] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the embodiments of the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] Embodiments of the present application are described with reference to the flowcharts and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate means for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0095] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0096] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0097] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic creative concepts. Therefore, the appended claims are intended to be construed as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0098] Finally, it should also be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or terminal device including the said element.

[0099] The above has introduced in detail a composite current limiting method and system for large flow data transmission based on the digital networking. Specific examples are used in this article to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A composite current limiting method for high-volume data transmission based on digital networking, characterized in that: The method comprises: Parse the protocol header of the request data to determine the target communication protocol for data transmission; Determining a target data type and a priority value of the digital object according to metadata in the requested digital object; Determine a corresponding first current limiting strategy according to the target communication protocol and the target data type; According to the priority value, modify the parameters of the first current limiting strategy to obtain a second current limiting strategy; The digital object is subjected to current limiting transmission by using the second current limiting strategy.

2. According to claim 1, a composite current limiting method for large-volume data transmission based on digital networking is characterized in that: According to the priority value, modifying the parameters of the first current limiting strategy to obtain a second current limiting strategy includes: Determining target parameters for parameter correction according to the first current limiting strategy; According to the priority value and the preset correction rule, the target parameter is corrected to obtain a second current limiting strategy.

3. According to claim 2, a composite current limiting method for large-volume data transmission based on digital networking is characterized in that: According to the priority value and the preset correction rule, the target parameter is corrected to obtain a second current limiting strategy, including: Determine a basic priority value according to the priority values ​​of all digital objects in the cache queue; Determining a priority deviation value between a priority value of the digital object and the basic priority value; The target parameter is modified according to the priority deviation value, the base value and adjustment step of the target parameter, and the target percentage value to obtain a second current limiting strategy, wherein the target percentage value is a value corresponding to both the priority value of the digital object and the type of the target parameter.

4. According to claim 1, a composite current limiting method for large-volume data transmission based on digital networking is characterized in that: Before modifying the parameters of the first current limiting strategy according to the priority value to obtain the second current limiting strategy, the method further includes: Adjusting the priority value of the digital object according to the waiting time of the digital object in the cache queue; The step of modifying parameters of the first current limiting strategy according to the priority value to obtain a second current limiting strategy includes: According to the adjusted priority value, the first current limiting strategy is parameter modified to obtain the second current limiting strategy.

5. According to claim 1, a composite current limiting method for large-volume data transmission based on digital networking is characterized in that: Before performing current limiting transmission on the digital object by using the second current limiting strategy, the method further includes: Determine actual measurements of network traffic during data transmission; According to the actual measured value and the set value of the network flow, dynamically adjusting the second current limiting strategy through PID feedback regulation; The step of performing current limiting transmission on the digital object by using the second current limiting strategy includes: The digital object is subjected to current-limited transmission using the dynamically adjusted second current-limiting strategy.

6. A composite current limiting method for high-volume data transmission based on digital networking according to claim 5, characterized in that: According to the actual measured value and the set value of the network flow, the second current limiting strategy is dynamically adjusted through PID feedback regulation, including: In the case where the second current limiting strategy is a token bucket algorithm, dynamically adjusting the token bucket capacity or token generation rate in the second current limiting strategy through PID feedback regulation according to the actual measurement value and the set value of the network traffic; In the case where the second current limiting strategy is a leaky bucket algorithm, dynamically adjusting the leaky bucket capacity or water outflow rate in the second current limiting strategy through PID feedback regulation according to the actual measurement value and the set value of the network flow; When the second current limiting strategy is a sliding window algorithm, the window size or request processing rate in the second current limiting strategy is dynamically adjusted through PID feedback regulation according to the actual measurement value and the set value of the network traffic.

7. A composite current limiting method for high-volume data transmission based on digital networking according to claim 4, characterized in that: The method further comprises: When the priority values ​​of all digital objects in the cache queue are adjusted to the highest priority, resources are evenly allocated for data transmission of each digital object.

8. The composite current limiting method for high-volume data transmission based on digital networking according to claim 1 is characterized in that: In the case where the target communication protocol is the HTTP protocol, the method further includes: Filter the request data through a regular rule library to determine whether the request data belongs to a known attack pattern; In the case of no rule hit, the parameter semantics of the request data are analyzed by the AST parsing engine to determine whether the request data is a suspicious request; If the request data is a suspicious request, direct the request data to an isolated environment for execution to monitor abnormal behavior; If no abnormal behavior is detected, respond to the request data.

9. The composite current limiting method for high-volume data transmission based on digital networking according to claim 1 is characterized in that: In the case where the target communication protocol is the gRPC protocol, the method further includes: Verify the RPC method signature in the request data to determine whether the request data is legal, wherein the RPC method signature is a unique signature generated by the server for each RPC method; If the request data is legal, determining whether the call chain of the request data is an abnormal call chain according to the request data and the service dependency graph; In the case where the call chain is an abnormal call chain, the abnormal call chain is blocked, and the service dependency graph is a dependency relationship between services constructed based on information recorded by each data request; or Determine whether the request data is legal according to a dynamic token in the request data, wherein the dynamic token is sent to the client after the server verifies the certificate and fingerprint of the connection request of the client sending the request data; If the request data is legal, respond to the request data.

10. The composite current limiting method for high-volume data transmission based on digital networking according to claim 1 is characterized in that: In the case where the target communication protocol is the WebSocket protocol, the method further includes: Performing a binary magic number check on the request data to determine whether the request data is legal, and / or performing a malicious keyword scan on the request data to determine whether the request data is legal; If the request data is legal, respond to the request data; Adjust the number of connections for each client based on the server's CPU usage; According to a preset heartbeat monitoring mechanism, a request connection that complies with the preset heartbeat monitoring mechanism is disconnected.

11. A composite current limiting system for high-volume data transmission based on a digital network, characterized in that: The system comprises: A target communication protocol determination module, used to parse the protocol header of the request data and determine the target communication protocol for data transmission; A metadata parsing module, used to determine the target data type and priority value of the digital object according to the metadata in the requested digital object; A first current limiting strategy determining module, used to determine a corresponding first current limiting strategy according to the target communication protocol and the target data type; A second current limiting strategy determination module is used to modify the parameters of the first current limiting strategy according to the priority value to obtain a second current limiting strategy; The current limiting control module is used to perform current limiting transmission on the digital object through the second current limiting strategy.

Images