Data transmission bandwidth limiting method and system, storage medium and equipment

Abstract

The invention provides a data transmission bandwidth limiting method and system, a storage medium and equipment. The method comprises the following steps: acquiring real-time characteristics, performing intelligent scheduling according to the real-time characteristics, and generating a scheduling decision; and sending data based on the scheduling decision. The data transmission bandwidth limitation of the present invention improves the availability and integrity of online services.

Description

Technical Field

[0001] The present invention belongs to the technical field of data transmission, and in particular relates to a data transmission bandwidth limitation method, system, storage medium and device. Background Art

[0002] With the gradual deepening of information security in recent years, private networks are needed to connect sensitive data and user privacy data between different applications. However, under the condition of limited bandwidth of private networks, it is particularly important to maximize their role. Therefore, it is necessary to manage the applications using the private network intelligently to achieve high bandwidth availability to ensure maximum utilization under limited resources.

[0003] Currently, the management technology of bandwidth resources is to monitor the bandwidth dimension. However, if the bandwidth usage is large, data packets will be lost during peak hours, or resource competition between services will cause services to be unavailable during certain peak hours, reducing the availability of services and bringing security risks to online services. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a method, system, storage medium and device for limiting data transmission bandwidth. Based on real-time characteristics, intelligent scheduling is performed through a deep model to decide when and how the requested data is sent. Data transmission can be intelligently scheduled and adjusted in real time to improve the availability and integrity of online services.

[0005] The present invention is achieved through the following technical solutions:

[0006] Acquire real-time features, perform intelligent scheduling based on the real-time features, and generate scheduling decisions;

[0007] Based on the scheduling decision, data is sent.

[0008] Optional,

[0009] The obtaining of real-time features includes:

[0010] Collect application latency characteristics, current bandwidth indicators, and I / O performance characteristics.

[0011] Optional,

[0012] The method further comprises:

[0013] Flink is used to process the data domain features of application monitoring, network monitoring, and host monitoring in real time to obtain real-time features.

[0014] Optional,

[0015] The intelligent scheduling according to the real-time characteristics and generating a scheduling decision include:

[0016] The real-time features are called, intelligent scheduling is performed through the deep model, and scheduling decisions are calculated and generated.

[0017] Optional,

[0018] The intelligent scheduling through the deep model and the calculation and generation of scheduling decisions include:

[0019] Input the current real-time bandwidth characteristic information, the remaining time of the request, and the size of the data to be sent this time to the lightweight network composed of multiple layers of MLP networks;

[0020] Input the real-time features into a complex network consisting of an Embedding layer and multiple layers of TransFormer;

[0021] The output results of the lightweight network and the complex network are spliced ​​together, and the scheduling decision is generated through the MLP layer.

[0022] Optional,

[0023] The sending of data based on the scheduling decision includes:

[0024] Based on the scheduling decision, the sender generates a message;

[0025] Determine whether the message size exceeds a preset threshold:

[0026] If the size of the message exceeds a preset threshold, the message is split, compressed and the data is sent;

[0027] If the message size does not exceed a preset threshold, the message is directly compressed and the data is sent.

[0028] Optional,

[0029] The method further comprises:

[0030] The receiving end receives data;

[0031] Determine whether the data received by the receiving end is compressed, and if the data received by the receiving end is compressed, decompress it according to the corresponding compression algorithm;

[0032] It is determined whether the decompressed message is split, and if the decompressed message is split, the decompressed message is assembled.

[0033] The present invention also provides a data transmission bandwidth limitation system for implementing the above method, the system comprising:

[0034] An intelligent scheduling module, used to obtain real-time features, perform intelligent scheduling according to the real-time features, and generate scheduling decisions;

[0035] A data sending module is used to send data based on the scheduling decision.

[0036] Optional,

[0037] The intelligent scheduling module is also configured to:

[0038] Flink is used to process the data domain features of application monitoring, network monitoring, and host monitoring in real time to obtain real-time features.

[0039] Optional,

[0040] The intelligent scheduling module is also configured to:

[0041] The real-time features are called, and intelligent scheduling is performed through the deep model to calculate and generate scheduling decisions, including:

[0042] Input the current real-time bandwidth characteristic information, the remaining time of the request, and the size of the data to be sent this time to the lightweight network composed of multiple layers of MLP networks;

[0043] Input the real-time features into a complex network consisting of an Embedding layer and multiple layers of TransFormer;

[0044] The output results of the lightweight network and the complex network are spliced ​​together, and the scheduling decision is generated through the MLP layer.

[0045] Optional,

[0046] The data sending module is configured to:

[0047] Based on the scheduling decision, the sender generates a message;

[0048] Determine whether the message size exceeds a preset threshold:

[0049] If the size of the message exceeds a preset threshold, the message is split, compressed and the data is sent;

[0050] If the message size does not exceed a preset threshold, the message is directly compressed and the data is sent.

[0051] Optional,

[0052] The system is also configured to:

[0053] The receiving end receives data;

[0054] Determine whether the data received by the receiving end is compressed, and if the data received by the receiving end is compressed, decompress it according to the corresponding compression algorithm;

[0055] It is determined whether the decompressed message is split, and if the decompressed message is split, the decompressed message is assembled.

[0056] The present invention also provides a computer-readable storage medium storing one or more programs, which, when executed, can implement the aforementioned data transmission bandwidth limitation method.

[0057] The present invention also provides a device, comprising a processor, a communication interface, a computer-readable storage medium and a communication bus; wherein the processor, the communication interface and the computer-readable storage medium communicate with each other via the communication bus;

[0058] The processor is configured to execute a program stored in a computer-readable storage medium.

[0059] Compared with the prior art, the present invention has the following advantages:

[0060] 1. The data transmission bandwidth limitation method proposed in the present invention can maximize the utilization of bandwidth resources under the condition of limited network bandwidth and improve the availability of dedicated line services by introducing a deep model for real-time intelligent application and bandwidth management based on real-time characteristics.

[0061] 2. Use deep models to perform intelligent and refined resource management and data transmission, obtain real-time features through real-time feature processing, and provide a data basis for the intelligent deep model to accurately judge scheduling tasks, so that the deep model can perform optimal scheduling more accurately and in real time, thereby improving the accuracy of the deep model.

[0062] 3. The deep model network ensures the calculation speed through a lightweight network, and ensures the best model effect through a complex network. The deep model network structure minimizes the time consumed by intelligent scheduling calculations while ensuring the best recommendation, and has almost no delay impact on online service requests.

[0063] 4. Using deep models for intelligent and refined resource management and data transmission, scheduling and adjustment can be performed intelligently in real time. The entire process is intelligently scheduled and managed without human involvement.

[0064] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 A schematic diagram of a flow chart of a data transmission bandwidth limitation method is shown;

[0067] Figure 2 A structural schematic block diagram of a data transmission bandwidth limitation system is shown;

[0068] Figure 3 A schematic diagram of a method for limiting bandwidth of cross-airport data transmission according to an embodiment of the present invention is shown;

[0069] Figure 4 A schematic diagram of a message compression transmission process according to an embodiment of the present invention is shown;

[0070] Figure 5 A schematic diagram of the overall intelligent scheduling process according to an embodiment of the present invention is shown;

[0071] Figure 6 A schematic diagram of the deep model network structure of an embodiment of the present invention is shown;

[0072] Figure 7 A schematic diagram of a large message processing flow in an embodiment of the present invention is shown;

[0073] Figure 8 It is a structural schematic diagram of a device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] See attached Figure 1 , the method of the present invention comprises:

[0076] S1. Acquire real-time features, perform intelligent scheduling according to the real-time features, and generate scheduling decisions;

[0077] The real-time characteristics acquisition includes: collecting application delay characteristics, current bandwidth indicators and I / O performance characteristics,

[0078] Among them, Flink is used to process the data domain features of application monitoring, network monitoring, and host monitoring in real time to obtain real-time features.

[0079] The real-time features are called, intelligent scheduling is performed through the deep model, and scheduling decisions are generated by calculation.

[0080] Among them, the current real-time bandwidth characteristic information, the remaining time of the request and the size of the data sent this time are input into the lightweight network composed of multiple layers of MLP network;

[0081] Input the real-time features into a complex network consisting of an Embedding layer and multiple layers of TransFormer;

[0082] The output results of the lightweight network and the complex network are spliced ​​together, and the scheduling decision is generated through the MLP layer.

[0083] S2. Send data based on the scheduling decision.

[0084] Wherein, based on the scheduling decision, the sending end generates a message;

[0085] Determine whether the message size exceeds a preset threshold:

[0086] If the size of the message exceeds a preset threshold, the message is split, compressed and the data is sent;

[0087] If the message size does not exceed the preset threshold, the message is directly compressed and the data is sent.

[0088] Wherein, the receiving end receives the data sent by the sending end;

[0089] Determining whether the received data is compressed, and if the data is compressed, decompressing it according to a corresponding compression algorithm;

[0090] It is determined whether the decompressed message is split, and if the message is split, the message is assembled.

[0091] For example,

[0092] The embodiment of the present invention uses a deep model to perform intelligent scheduling based on real-time features to decide when and how to send the requested data. It can intelligently schedule and adjust data transmission in real time to improve the availability and integrity of online services.

[0093] 1. Intelligent scheduling.

[0094] In this embodiment, intelligent scheduling is performed according to the current bandwidth and system, I / O and other conditions, thereby meeting the availability of online services under bandwidth constraints. The core idea is to divide application requests that need to be transmitted across airports into two categories: high latency and low latency: high latency requests have low latency requirements; low latency requests are more sensitive to latency and need to return results within a specified time.

[0095] 1. Obtain real-time features.

[0096] In this embodiment, the real-time feature acquisition includes:

[0097] Collect application latency characteristics, current bandwidth indicators, and I / O performance characteristics.

[0098] Application latency characteristics, including application process data processing latency, residual latency for each request, application latency limit, etc.

[0099] Current bandwidth status indicators, including: used bandwidth, available bandwidth, bandwidth utilization, etc.

[0100] I / O performance characteristics, including I / O speed, packet size, high performance, and low performance latency in the last minute.

[0101] In this embodiment, the characteristics of data domains such as application monitoring, network monitoring, and host monitoring information can be analyzed and processed in real time through Flink streaming processing to obtain real-time features, providing a data basis for the intelligent deep model to accurately determine the scheduling tasks.

[0102] 2. Perform intelligent scheduling and generate scheduling decisions.

[0103] In this embodiment, the acquired real-time features are called, intelligent scheduling is performed through a deep model, and scheduling decisions are calculated and generated.

[0104] The deep model network is mainly divided into two parts: one part is a lightweight network composed of a multi-layer MLP network. This part mainly inputs the current real-time bandwidth feature information, the remaining time of the request, and the size of the data sent this time. The use of a simple multi-layer MLP can ensure that this part of the network is calculated quickly; the other part is a complex network composed of an Embedding layer and a multi-layer TransFormer. Through the statistical features of the host information, network information, application information, and the scheduled delay time of the request from the last minute, the model effect can be guaranteed to be the best. After splicing the output results of the two parts of the network, an MLP layer is used to complete the calculation and output of the final prediction result.

[0105] The deep model network structure in this embodiment can minimize the time consumed by intelligent scheduling calculations while ensuring the best scheduling decisions, and has almost no delay impact on online service requests.

[0106] In this embodiment, a deep model is used for intelligent and refined resource management and data transmission, and the deep model generates more accurate optimal scheduling decisions by processing real-time features in real time.

[0107] 2. Send data based on the scheduling decision.

[0108] Based on the scheduling decision, sending data includes: based on the scheduling decision, the sending end generates a message;

[0109] Determine whether the message size exceeds a preset threshold:

[0110] If the message size exceeds a preset threshold, the message is split;

[0111] If the message size does not exceed a preset threshold, the message is directly compressed and the data is sent.

[0112] In this embodiment, the message can be split according to the size of the message; and / or the message can be compressed.

[0113] Message splitting: When network data is transmitted across airports on a dedicated network, larger messages are split into multiple small messages to avoid the impact of large messages blocking the network.

[0114] Message compression: When network data is transmitted across airports on a dedicated network, a compression method is used to compress the message to reduce bandwidth usage.

[0115] In this embodiment, after the message is split, the size of each segmented message is equal to a preset segmentation threshold (configurable, 32KB by default), so except for the last segment of the message, the sizes of the remaining messages are equal; the receiving end will perform deduplication processing after receiving the message; the received message is judged whether it is a fragmented message based on whether the number of segments in the attribute header is null; the received fragmented message attribute headers with the same original message ID and airport three-letter code belong to the same large message (the original message ID of the same airport is unique); when assembling the message, it is judged whether the message is completely received based on the number of segments in the message attribute header being equal to the total number of received message fragments.

[0116] In this embodiment, the message will be compressed before sending, and the corresponding decompression will be performed after receiving the message. The message attribute header will indicate the type of compression algorithm, and the system currently mainly uses the zlib compression algorithm.

[0117] For the transmission of messages whose size exceeds the segmentation threshold (the threshold is configurable and the default value is 32KB), they will be split and compressed before sending. When receiving, the segmented messages will be decompressed and reassembled to obtain the original message.

[0118] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0119] See attached Figure 2 , shows the structure of a data transmission bandwidth limitation system for implementing the above method, including an intelligent scheduling module and a data sending module.

[0120] The intelligent scheduling module is used to obtain real-time features, perform intelligent scheduling according to the real-time features, and generate scheduling decisions;

[0121] The data sending module is used to send data based on the scheduling decision.

[0122] Figure 3 This is a schematic diagram of a method for limiting bandwidth for cross-airport data transmission according to an embodiment of the present invention. Data is transmitted from an application on host 1 to an application on host 2 over a dedicated network across airports. The core links include: intelligent scheduling, message splitting and message compression.

[0123] Figure 4 This is a schematic diagram of the message compression transmission process of an embodiment of the present invention. The sending end compresses the message and then sends the message; the receiving end determines whether the received message is compressed: if it is a compressed message, the message is decompressed according to the compression algorithm corresponding to the attribute, and the message is parsed; if it is an uncompressed message, the message is directly parsed.

[0124] Figure 5 This is a schematic diagram of the overall process of intelligent scheduling in an embodiment of the present invention. The data sending end sends a sending request to the intelligent scheduling service. The intelligent scheduling service calls the acquired real-time feature calculation to generate a decision result and sends data according to the decision result.

[0125] Among them, the relevant data features of application monitoring, network monitoring and host monitoring are formed into a message queue; Flink is used to process the real-time data in the message queue and extract real-time features; the extracted real-time features are converted so that the intelligent scheduling service can call them at any time.

[0126] Figure 6This is a schematic diagram of the deep model network structure of an embodiment of the present invention. The deep model network is mainly divided into two parts: the left part is a lightweight network composed of a multi-layer MLP network, and the input features are: current real-time bandwidth feature information, remaining request time, and the size of the data sent this time; the right part is a complex network composed of an Embedding layer and a multi-layer TransFormer, and the input features are: host information, network information, application information, and request scheduled delay time from the previous minute. After splicing the output results of the two parts of the network, the calculation and output of the final prediction result are completed through an MLP layer.

[0127] Figure 7 The figure is a schematic diagram of the large message processing flow according to an embodiment of the present invention.

[0128] After the sender generates a message, it determines whether the message size exceeds the threshold: if it exceeds the threshold, the message is split; if it does not exceed the threshold, the message is directly compressed and sent.

[0129] After receiving the message, the receiving end determines whether the message is compressed: if it is a compressed message, the message is decompressed according to the corresponding compression algorithm; if it is an uncompressed message, it is directly determined whether it is a segmented message: if it is a segmented message, the message is assembled and then parsed.

[0130] In addition, an embodiment of the present invention further provides a data transmission bandwidth limiting device, including:

[0131] The intelligent scheduling module obtains real-time features, performs intelligent scheduling based on the real-time features, and generates scheduling decisions; wherein, the data domain features of application monitoring, network monitoring, and host monitoring are processed in real time through Flink to obtain real-time features; the real-time features are called, and intelligent scheduling is performed through a deep model to calculate and generate scheduling decisions, including: inputting the current real-time bandwidth feature information, the remaining time of the request, and the size of the data sent this time into a lightweight network composed of a multi-layer MLP network; inputting the real-time features into a complex network composed of an Embedding layer and a multi-layer TransFormer; splicing the results output by the lightweight network and the complex network, and generating scheduling decisions through the MLP layer.

[0132] The data sending module sends data based on the scheduling decision. Based on the scheduling decision, the sending end generates a message and determines whether the message size exceeds a preset threshold: if the message size exceeds the preset threshold, the message is split and compressed and the data is sent; if the message size does not exceed the preset threshold, the message is directly compressed and the data is sent.

[0133] Based on the same inventive concept, the present invention also provides a computer-readable storage medium storing one or more programs, which, when executed, can implement the aforementioned data transmission bandwidth limitation method.

[0134] like Figure 8 As shown, an embodiment of the present invention further provides a device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus.

[0135] The memory is a computer-readable storage medium for storing one or more programs.

[0136] The processor is used to execute the program stored in the computer-readable storage medium.

[0137] The computer-readable storage medium may be included in the device / apparatus described in the above embodiments; or may exist independently without being assembled into the device / apparatus.

[0138] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data transmission bandwidth limiting method, characterized in that: include: Acquire real-time features, perform intelligent scheduling based on the real-time features, and generate scheduling decisions; Based on the scheduling decision, data is sent.

2. The method according to claim 1, characterized in that The obtaining of real-time features includes: Collect application latency characteristics, current bandwidth indicators, and I / O performance characteristics.

3. The method according to claim 2, characterized in that The method further comprises: Flink is used to process the data domain features of application monitoring, network monitoring, and host monitoring in real time to obtain real-time features.

4. The method according to claim 3, characterized in that The intelligent scheduling according to the real-time characteristics and generating a scheduling decision include: The real-time features are called, intelligent scheduling is performed through the deep model, and scheduling decisions are calculated and generated.

5. The method according to claim 4, characterized in that The intelligent scheduling through the deep model and the calculation and generation of scheduling decisions include: Input the current real-time bandwidth characteristic information, the remaining time of the request, and the size of the data to be sent this time to the lightweight network composed of multiple layers of MLP networks; Input the real-time features into a complex network consisting of an Embedding layer and multiple layers of TransFormer; The output results of the lightweight network and the complex network are spliced ​​together, and the scheduling decision is generated through the MLP layer.

6. The method according to claim 1, characterized in that The sending of data based on the scheduling decision includes: Based on the scheduling decision, the sender generates a message; Determine whether the message size exceeds a preset threshold: If the size of the message exceeds a preset threshold, the message is split, compressed and the data is sent; If the message size does not exceed a preset threshold, the message is directly compressed and the data is sent.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: The receiving end receives data; Determine whether the data received by the receiving end is compressed, and if the data received by the receiving end is compressed, decompress it according to the corresponding compression algorithm; It is determined whether the decompressed message is split, and if the decompressed message is split, the decompressed message is assembled.

8. A data transmission bandwidth limitation system, characterized in that: The system comprises: An intelligent scheduling module, used to obtain real-time features, perform intelligent scheduling according to the real-time features, and generate scheduling decisions; A data sending module is used to send data based on the scheduling decision.

9. The system according to claim 8, characterized in that The intelligent scheduling module is also configured to: Flink is used to process the data domain features of application monitoring, network monitoring, and host monitoring in real time to obtain real-time features.

10. The system according to claim 8, characterized in that The intelligent scheduling module is also configured to: The real-time features are called, and intelligent scheduling is performed through the deep model to calculate and generate scheduling decisions, including: Input the current real-time bandwidth characteristic information, the remaining time of the request, and the size of the data to be sent this time to the lightweight network composed of multiple layers of MLP networks; Input the real-time features into a complex network consisting of an Embedding layer and multiple layers of TransFormer; The output results of the lightweight network and the complex network are spliced ​​together, and the scheduling decision is generated through the MLP layer.

11. The system according to claim 8, characterized in that The data sending module is configured to: Based on the scheduling decision, the sender generates a message; Determine whether the message size exceeds a preset threshold: If the size of the message exceeds a preset threshold, the message is split, compressed and the data is sent; If the message size does not exceed a preset threshold, the message is directly compressed and the data is sent.

12. The system according to any one of claims 8 to 11, characterized in that: The system is also configured to: The receiving end receives data; Determine whether the data received by the receiving end is compressed, and if the data received by the receiving end is compressed, decompress it according to the corresponding compression algorithm; It is determined whether the decompressed message is split, and if the decompressed message is split, the decompressed message is assembled.

13. A computer-readable storage medium storing one or more programs, characterized in that: When the one or more programs are executed, the data transmission bandwidth limiting method described in any one of claims 1 to 7 can be implemented.

14. An electronic device comprising a processor, a communication interface, the computer-readable storage medium of claim 13, and a communication bus; wherein: The processor, the communication interface, and the computer-readable storage medium communicate electronically with each other via a communication bus; characterized in that: The processor is configured to execute a program stored in a computer-readable storage medium.

Images