Data transmission method and device

Abstract

The embodiment of the invention provides a data transmission method and device. The data transmission method comprises the steps that streaming data and batch data from different data sources are acquired; dividing the streaming data and the batch data into a plurality of priorities according to the importance degree and the data type, and storing the data of each priority in a corresponding priority queue; scheduling other priority queues except the highest priority queue according to a preset queue scheduling method to generate a virtual queue; scheduling the virtual queue and the highest priority queue according to a priority queue algorithm to obtain an output queue; and transmitting the data in the output queue. According to the invention, cooperative transmission of streaming data and batch data can be realized, the data transmission efficiency and reliability are improved, and the service quality is ensured.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular, to a data transmission method and device. Background Art

[0002] The State Grid Data Center is the core platform for data management and services. Through multi-source data fusion and data mining, it is conducive to business innovation and improves the overall operational efficiency of the power industry. Massive multi-source data supports a variety of application scenarios and plays an important role in power production and emergency decision-making.

[0003] In the data transmission process between the headquarters data center, the provincial data center, and the data centers of various business departments, there are a large amount of streaming data and batch data. Batch data has the transmission requirements of high throughput and low resource usage, while streaming data has the transmission requirements of low latency and high reliability. Therefore, how to coordinate the transmission of streaming data and batch data to ensure data transmission efficiency and service quality is a problem that needs to be solved. Summary of the invention

[0004] In view of this, the purpose of the embodiments of the present application is to propose a data transmission method and device to solve the problem of coordinated transmission of streaming data and batch data.

[0005] Based on the above purpose, an embodiment of the present application provides a data transmission method, including:

[0006] Get streaming and batch data from different data sources;

[0007] According to the importance and data type, the streaming data and batch data are divided into multiple priorities, and the data of each priority is stored in a corresponding priority queue;

[0008] For queues of other priorities except the highest priority queue, schedule them according to the preset queue scheduling method to generate virtual queues;

[0009] For the virtual queue and the highest priority queue, scheduling is performed according to the priority queue algorithm to obtain an output queue;

[0010] The data in the output queue is transmitted.

[0011] Optionally, the streaming data and batch data are divided into multiple priorities according to importance and data type, and the data of each priority is stored in a corresponding priority queue, including:

[0012] Important data that affects the operation of the power grid is regarded as the highest priority data and stored in the highest priority queue;

[0013] Streaming data that affects real-time analysis and decision-making of the power grid is treated as high-priority streaming data and saved in a high-priority stream queue;

[0014] The batch data of power finance is regarded as high-priority batch data and saved in the high-priority batch queue;

[0015] The power grid feedback streaming data is regarded as medium priority streaming data and saved in the medium priority stream queue;

[0016] The batch data of power grid design and operation and maintenance are regarded as medium priority batch data and stored in the medium priority batch queue;

[0017] Treat other data as low priority data and save them in the low priority queue.

[0018] Optionally, for queues of other priorities except the highest priority queue, scheduling is performed according to a preset queue scheduling method to generate a virtual queue, including:

[0019] Determining the dynamic weight of each queue in the other priority queues according to the priority, average delay and load parameters of each queue in the other priority queues;

[0020] Determining the number of polling times for each queue according to the dynamic weight;

[0021] Read data from each queue according to the polling times of each queue;

[0022] The virtual queue is constructed based on the read data.

[0023] Optionally, determining the dynamic weight of each queue in the other priority queues according to the priority, average delay, and load parameter of each queue in the other priority queues includes:

[0024] For any queue in each queue, a weight adjustment factor is calculated according to the priority, average delay and load parameters of the queue;

[0025] The weight of the queue in the next scheduling period is determined according to the weight of the current scheduling period and the weight adjustment factor.

[0026] Optionally, a weight adjustment factor is calculated according to the priority, average delay, and load parameters of the queue, as follows:

[0027] ΔW i =αP i +βDF i +γQF i (2)

[0028] Where, ΔW i is the weight adjustment factor of the i-th queue, Pi is the priority of the i-th queue, DF i is the average delay of the ith queue, QF i is the load parameter of the ith queue, and α, β, and γ are weight coefficients.

[0029] Optionally, the weight of the queue in the next scheduling period is determined according to the weight of the current scheduling period and the weight adjustment factor, and the method is:

[0030] W i (t+1) =W i (t) +ΔW i (8)

[0031] Among them, W i (t) is the weight of the i-th queue in the t-th scheduling cycle, W i (t+1) is the weight of the i-th queue in the t+1-th scheduling period.

[0032] Optionally, the polling times of each queue are determined according to the dynamic weight, and the method is:

[0033]

[0034] Among them, N i is the polling times of the i-th queue in the next scheduling cycle, W i is the weight of the i-th queue in the next scheduling cycle, n is the total number of queues of other priority levels, and N is the preset total number of polling times.

[0035] Optionally, transmitting the data in the output queue includes:

[0036] Selecting an adapted target link from a pre-constructed link list according to quality requirements of the streaming data and / or batch data to be transmitted in the output queue;

[0037] The target link is used to transmit the streaming data and / or batch data to be transmitted.

[0038] Optionally, transmitting the streaming data and / or batch data to be transmitted by using the target link includes:

[0039] Sending a test data packet for detecting the link status to the target link according to a preset period;

[0040] Determine whether to switch the target link according to the test result of the test data packet.

[0041] The present application also provides a data transmission device, including:

[0042] The acquisition module is used to obtain streaming data and batch data from different data sources;

[0043] A queue construction module, used to divide the streaming data and batch data into multiple priorities according to importance and data type, and store the data of each priority in a corresponding priority queue;

[0044] A first scheduling module, used for scheduling other priority queues except the highest priority queue according to a preset queue scheduling method to generate a virtual queue;

[0045] A second scheduling module is used to schedule the virtual queue and the highest priority queue according to the priority queue algorithm to obtain an output queue;

[0046] A transmission module is used to transmit the data in the output queue.

[0047] As can be seen from the above, the data transmission method and device provided by the embodiment of the present application, by obtaining streaming data and batch data from different data sources, dividing the streaming data and batch data into multiple priorities according to the importance and data type, storing the data of each priority in the corresponding priority queue, and scheduling according to the preset queue scheduling method for other priority queues except the highest priority queue, generating a virtual queue, and scheduling according to the priority queue algorithm for the virtual queue and the highest priority queue, obtaining an output queue, and transmitting the data in the output queue. The present application can realize the coordinated transmission of streaming data and batch data, improve the efficiency and reliability of data transmission, and ensure the quality of service. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 A schematic diagram of a method flow of an embodiment of the present application;

[0050] Figure 2 A schematic diagram of a queue scheduling process according to an embodiment of the present application;

[0051] Figure 3 A schematic diagram of the data transmission process of an embodiment of the present application;

[0052] Figure 4 A schematic diagram of the device structure of an embodiment of the present application;

[0053] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0055] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connecting" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to represent relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0056] As mentioned in the background technology section, there are streaming data and batch data in the data center, and the problem of coordinated transmission of the two types of data needs to be solved. In the related technology, data transmission can be achieved based on the queue scheduling method. However, the relevant queue scheduling methods have their own characteristics, and there is no effective method suitable for the coordinated transmission of streaming data and batch data in the data center.

[0057] In view of this, an embodiment of the present application provides a data transmission method, which obtains streaming data and batch data from different data sources, divides the priorities of streaming data and batch data according to importance and data type, and puts data of different priorities into queues of corresponding priorities. For priority queues other than the highest priority queue, scheduling is performed according to a preset queue scheduling method to generate a virtual queue. According to the priority queue algorithm, the virtual queue and the highest priority queue are scheduled to obtain an output queue, and then the data in the output queue is transmitted. This method can comprehensively consider priority and fairness to achieve coordinated transmission of streaming data and batch data.

[0058] The technical solution of the present application is further described in detail below through specific embodiments.

[0059] like Figure 1 As shown, an embodiment of the present application provides a data transmission method, including:

[0060] S101: Obtain streaming data and batch data from different data sources;

[0061] In this embodiment, the data middle platform receives data from different data sources, including data from data middle platforms of different levels divided according to regional levels, data from data middle platforms of various departments, data from various business systems (such as billing systems, operation and maintenance systems, monitoring systems, etc.), etc.

[0062] Among them, data can be divided into streaming data and batch data. Streaming data is data that is generated in real time and needs to be continuously transmitted. The data volume is small and the data quality is relatively low. For example, real-time monitoring data generated by smart meters, transformers and other equipment, as well as real-time analysis data generated by video surveillance and voice recognition applications. Batch data is generated periodically or irregularly, transmitted once and has a large data volume. It has the characteristics of high reliability and low real-time requirements. For example, statistical reports such as daily and monthly reports generated by power market and financial management business systems, as well as offline analysis results generated by artificial intelligence and big data mining applications. This embodiment does not provide detailed examples and limitations on the specific data types and contents of the data center.

[0063] S102: Divide the streaming data and batch data into multiple priorities according to the importance and data type, and store the data of each priority in a corresponding priority queue;

[0064] In this embodiment, considering that various types of data in the data platform have different degrees of influence on the operation of the power grid, and the transmission requirements of various types of data are different, after receiving the data, the streaming data and batch data are divided into multiple priorities according to the importance and data type of the data, and the data of each priority is stored in the corresponding priority queue. The method includes:

[0065] Important data that affects the operation of the power grid is regarded as the highest priority data and stored in the highest priority queue;

[0066] Streaming data that affects real-time analysis and decision-making of the power grid is treated as high-priority streaming data and saved in a high-priority stream queue;

[0067] The batch data of power finance is regarded as high-priority batch data and saved in the high-priority batch queue;

[0068] The power grid feedback streaming data is regarded as medium priority streaming data and saved in the medium priority stream queue;

[0069] The batch data of power grid design and operation and maintenance are regarded as medium priority batch data and stored in the medium priority batch queue;

[0070] Treat other data as low priority data and save them in the low priority queue.

[0071] In this embodiment, data is divided into six priorities according to the importance and data type of the data. As shown in Table 1, specifically, important data that affects the operation of the power grid is regarded as the highest priority data (priority H), such as power grid fault detection, safety protection, emergency control and other data that directly affect the operation of the power grid, which are of extremely high importance and urgency, require immediate response, and cannot be delayed. The highest priority data is stored in the highest priority queue and has the highest priority during data transmission. Streaming data that affects real-time analysis and decision-making of the power grid is regarded as high-priority streaming data (priority G), such as real-time analysis and decision-making data generated by applications such as power load forecasting, power quality monitoring, and intelligent scheduling, which are of high importance and real-time nature and require rapid processing and response. The high-priority streaming data is stored in the high-priority streaming queue. Batch data of power finance is regarded as high-priority batch data, such as offline reports and result data generated by applications such as power market transactions, financial management, and audit supervision, which have high throughput and low resource occupancy transmission requirements and require timely response. The high-priority batch data is stored in the high-priority batch queue. Streaming data of power grid feedback is treated as medium-priority streaming data (priority is M). For example, data of real-time feedback and suggestions such as power grid equipment status monitoring, energy management, and user services have certain importance and real-time requirements, and can tolerate a certain degree of delay. Medium-priority streaming data is stored in the medium-priority stream queue. Batch data of power grid design and operation and maintenance are treated as medium-priority batch data, such as offline analysis and evaluation data such as power grid planning and design, operation and maintenance optimization, and knowledge discovery. They have certain importance and timeliness, can tolerate resource occupation, and do not require timely processing and response. Medium-priority batch data is stored in the medium-priority batch queue. Other data is treated as low-priority data (priority is L), such as power grid historical data, backup data, log data, and other data that do not affect power grid operation, have low importance and real-time requirements, and can tolerate delays. Low-priority data is stored in the low-priority queue.

[0072] Table 1 Data priority classification

[0073]

[0074] S103: For queues of other priorities except the highest priority queue, schedule them according to a preset queue scheduling method to generate a virtual queue;

[0075] In this embodiment, after the acquired data is prioritized and corresponding priority queues are constructed, except for the highest priority queue, other priority queues are scheduled according to a preset queue scheduling method to generate virtual queues including data in other priority queues.

[0076] In some embodiments, scheduling is performed according to a preset queue scheduling method to generate a virtual queue, including:

[0077] Determine the dynamic weight of each queue in other priority queues according to the priority, average delay and load parameters of each queue in other priority queues;

[0078] Determine the polling times for each queue based on the dynamic weight;

[0079] Read data from each queue according to the polling times of each queue;

[0080] Build a virtual queue based on the read data.

[0081] In this embodiment, considering that in the weighted round robin scheduling algorithm (WRR, Weighted Round Robin), each queue is polled and scheduled according to a fixed weight, the priority difference of each queue and the actual change of the queue are not considered, which affects the fairness of data transmission of each queue. Therefore, the present application comprehensively considers the priority and actual change of each queue in other priority queues, and proposes an improved weighted round robin scheduling algorithm, which is used for scheduling, and dynamically adjusts the weight of each queue according to the actual change of the queue priority and average delay, load and other factors, determines the polling number of each queue according to the dynamic weight, and polls and schedules the data in each queue according to the polling number, and saves the data read out by scheduling in the virtual queue.

[0082] The improved weight-based round-robin scheduling algorithm model proposed in this embodiment can be expressed as:

[0083]

[0084] In the model, Q1, Q2, …, Q n-1 For all priority queues except the highest priority queue, Q n The highest priority queue, WQ, is a virtual queue constructed by scheduling other priority queues according to dynamic weights using an improved weight-based round-robin scheduling algorithm (also known as the DWRR algorithm); PQ is a virtual queue constructed by scheduling the virtual queue WQ and the highest priority queue Q n The output queue is generated after priority scheduling according to the PQ priority queue algorithm.

[0085] In some implementations, determining the dynamic weight of each queue in other priority queues according to the priority, average delay, and load parameters of each queue in other priority queues includes:

[0086] For any queue in each queue, a weight adjustment factor is calculated according to the priority, average delay and load parameters of the queue;

[0087] The weight of the queue in the next scheduling period is determined based on the weight of the current scheduling period and the weight adjustment factor.

[0088] The weight adjustment factor is calculated based on the queue priority, average delay, and load parameters as follows:

[0089] ΔW i =αP i +βDF i +γQF i (2)

[0090] Where, ΔW i is the weight adjustment factor of the i-th queue; P i is the priority of the i-th queue. Each priority queue can preset a corresponding priority value. For example, m queues are sorted from low to high in priority, and the priority values ​​can be assigned to 1-m respectively. DF i is the average delay of the ith queue, QF i is the load parameter of the i-th queue, α, β and γ are weight coefficients used to adjust the influence of different parameters. The larger the weight adjustment factor calculated by the above formula, the more the weight increases, the higher the transmission urgency of the queue, and the queue should be given priority.

[0091] In some embodiments, the maximum allowable delay of data refers to the maximum delay that can be tolerated by the data during transmission, and the actual waiting delay of data refers to the time interval from the generation to the transmission of data, reflecting the actual delay of the data during transmission. According to the delay requirement and actual delay of the data, a delay factor is assigned to each data, and the average value of the delay factors of all data in the queue reflects the delay sensitivity of the queue. Among them, the delay factor can be determined according to the ratio of the maximum allowable delay of the data to the actual waiting delay, expressed as:

[0092]

[0093] Among them, T j is the delay factor of the jth data, MD j is the maximum allowable delay of the jth data, WD j is the actual waiting delay of the jth data.

[0094] Then, according to the delay factor of the data, the average delay of the queue storing the data is determined as follows:

[0095]

[0096] Among them, DF i is the average delay of the ith queue, n i is the number of data in the i-th queue.

[0097] The load of a queue is determined by the queue's minimum reserved traffic rate (MRTR), average packet size, and remaining queue buffer length. According to the load of queues of different priorities, a load parameter is assigned to each data, indicating the degree of congestion in the queue where the data is located. The minimum reserved traffic rate refers to the minimum transmission requirement of the queue, the average packet size refers to the average length of the data in the queue, and the remaining queue buffer length refers to the length of the data that can be stored in the queue, which depends on the initial capacity of the queue and the length of the data already stored. The parameters can be expressed as:

[0098]

[0099] Among them, PS i is the average packet size of the ith queue, L j is the length of the jth data; RL i is the remaining queue buffer length of the i-th queue, CL i Represents the initial capacity of the i-th queue; MRTR i is the minimum congestion retention rate of the ith queue, QF j is the load parameter of the i-th data.

[0100] In some embodiments, the method for determining the weight of the queue in the next scheduling period according to the weight and weight adjustment factor of the current scheduling period is:

[0101] W i (t+1) =W i (t) +ΔW i (8)

[0102] Among them, W i (t) is the weight of the i-th queue in the t-th scheduling cycle, W i (t+1) is the weight of the i-th queue in the t+1-th scheduling cycle, that is, the weight of the next scheduling cycle is determined according to the weight of the current scheduling cycle and the weight adjustment factor determined in the current scheduling cycle. The larger the dynamically determined weight is, the more urgent the transmission of the queue is, and the priority service should be given. By comprehensively considering the priority of the queue and dynamically changing parameters such as load and delay, the weight is dynamically adjusted to adapt to the changing characteristics of the power grid batch data, taking into account the priority of the data and the fairness of data transmission.

[0103] In some methods, the polling times of each queue are determined according to the dynamic weight, and the method is as follows:

[0104]

[0105] Among them, N i is the polling times of the i-th queue in the next scheduling cycle, W i is the weight of the i-th queue in the next scheduling cycle, W j is the weight of the jth queue in the next scheduling cycle, n is the total number of queues of other priority levels, that is, the number of queues except the highest priority queue, and N is the preset total number of polling times.

[0106] In the present embodiment, for other priority queues except the highest priority queue, dispatch according to the improved polling scheduling algorithm based on weight, in each scheduling cycle, according to the priority, average delay and load parameter of each queue, determine the weight adjustment factor, determine the weight of next scheduling cycle according to the weight adjustment factor and the weight of current scheduling cycle, calculate the polling number of next scheduling cycle according to the determined weight, after entering next scheduling cycle, read data from corresponding queue according to the polling number of each queue, and the data read out are stored in the virtual queue.Like this, the virtual queue constructed considers the priority of each queue and the importance determined according to various parameters such as the delay, load of dynamic change simultaneously, guarantees the fairness of data transmission, can improve data transmission efficiency and service quality.Wherein, the data in the same queue adopts the first-in-first-out mode to read data.

[0107] S104: For the virtual queue and the highest priority queue, scheduling is performed according to the priority queue algorithm to obtain an output queue;

[0108] S105: Transmit the data in the output queue.

[0109] In the present embodiment, after the virtual queue is generated, for the virtual queue and the highest priority queue, the data in the virtual queue and the highest priority queue are scheduled according to the priority queue algorithm (PQ, Priority queuing), that is, as long as there is data in the highest priority queue, the data in the highest priority queue is preferentially read, and the read data is stored in the output queue for preferential transmission, and after all the data in the highest priority queue are read, the data in the virtual queue is read again, and the read data is stored in the output queue. The present embodiment does not elaborate on the specific principle and process of the priority queue algorithm.

[0110] like Figure 2As shown, in some implementations, the process of using the queue scheduling algorithm to process the data to be transmitted is: initialize the queue parameters such as the maximum length, current length, priority, dynamic weight, etc. of each queue; when new data arrives, save the data in the corresponding priority queue according to the priority of the data. If the queue is full, discard the data, and update the queue parameters after saving; for other priority queues, use the DWRR scheduler to schedule according to the improved queue scheduling algorithm to generate a virtual queue; for the highest priority queue and the virtual queue, use the PQ scheduler to schedule according to the priority queue algorithm to generate an output queue.

[0111] In some embodiments, considering that there are streaming data and batch data with different priorities and different transmission requirements in the output queue, if the same link is used to transmit various types of data in the output queue, the transmission efficiency and resource utilization will be affected. Therefore, in this embodiment, the method for transmitting data in the output queue includes:

[0112] Selecting an adapted target link from a pre-built link list according to quality requirements of streaming data and / or batch data to be transmitted in an output queue;

[0113] The target link is used to transmit the streaming data and / or batch data to be transmitted.

[0114] like Figure 3 As shown, in this embodiment, the output queue stores the streaming data and / or batch data to be transmitted. When transmitting data, according to the type and quality requirements of the data in the output queue, a target link that is adapted to the data to be transmitted is selected from the pre-built link list, and the data to be transmitted is transmitted using the target link. For streaming data and / or batch data of different priorities in the output queue, appropriate links are allocated for transmission, thereby improving data transmission efficiency and resource utilization and meeting the quality requirements of different data.

[0115] In some embodiments, the method for constructing a link list includes: sending a predetermined number of test data packets to each available link, and the test data packets are consistent with the transmission requirements of the data to be transmitted; determining the link's available bandwidth, delay and other indicators based on the test results of the test data packets, such as the transmission time and packet loss rate; determining the priority of each link based on the indicators of each link, and sorting the links in descending order according to the priority of each link, and generating a link list based on the sorted links, wherein links with high priorities have higher indicators and are more suitable for transmitting data.

[0116] In some methods, let the link set be L, define the link evaluation function and link priority, expressed as:

[0117] EF p =ω1B p -ω2DL p-ω3PL p (10)

[0118]

[0119] Among them, B p is the available bandwidth of link p, DL p is the delay of link p, PL p is the packet loss rate of link p, ω1, ω2, and ω3 are weight coefficients, which are adjusted according to different data types and QoS requirements. Link evaluation function EF p Reflects the performance of link p, link priority PR p is the priority of the link determined according to the performance of link p, EF q is the priority of link q, and K is the number of links.

[0120] For the streaming data and / or batch data to be transmitted in the output queue, an adapted target link is selected from the link list, including: for streaming data with high real-time requirements, the minimum delay strategy is adopted, and the link with the lowest delay is selected first; for batch data with high reliability requirements, the maximum bandwidth strategy is adopted, and the link with the largest bandwidth is selected to meet its high throughput requirements; for other types of streaming batch data, the minimum load strategy is adopted, and the link with the lowest load is allocated to balance the transmission efficiency and service quality of various types of data. In this way, according to the type of data to be transmitted and the transmission requirements, the appropriate link is allocated, which can realize the coordinated transmission of streaming data and batch data, meet the quality requirements of various types of data, and improve the efficiency of data transmission.

[0121] In some approaches, the load of link p can be calculated by The smaller the calculated result is, the lower the link load is and the idler the link is.

[0122] In some embodiments, transmitting the streaming data and / or batch data to be transmitted using the target link includes:

[0123] Sending a test data packet for detecting the link status to the target link according to a preset period;

[0124] Determine whether to switch the target link based on the test result of the test data packet.

[0125] In this embodiment, considering that the link status may change due to factors such as the environment, in order to ensure the quality of data transmission and adapt to changes in the network environment, it is necessary to dynamically monitor the link status during the process of transmitting data using the target link so that when the performance of the target link decreases, the link can be switched in time to avoid affecting data transmission.

[0126] Specifically, in the process of transmitting data using the target link, test data packets are periodically sent to the target link. The current performance of the target link is determined based on test results such as the transmission time and packet loss rate of the test data packets. If the current performance of the target link cannot meet the data transmission requirements, the link needs to be switched. When switching, a suitable link can be reselected from the link list.

[0127] In some methods, when a target link failure is detected, such as link interruption, severe congestion, etc., an emergency switching strategy is adopted to select the available link with the highest priority from the link list for switching to ensure data reliability and continuity. When it is detected that the link performance has dropped to more than the preset jitter limit parameter, such as the link delay increases to a certain extent, the bandwidth decreases to a certain extent, etc., an optimized switching strategy is adopted to select an available link with better performance than the current target link from the link list, and a certain proportion of data is transferred to the selected available link according to the data flow batch type and quality requirements to ensure data efficiency and accuracy. When link resources are detected to be idle, such as the link load is too low, the bandwidth is excessive, etc., a balanced switching strategy is adopted to transfer part of the data of other links to this link to improve the utilization and balance of the link. By detecting the link status in real time and using an appropriate link switching strategy for switching, it is possible to respond to link failures, performance degradation and idle resources in a timely manner to ensure the efficiency and stability of data transmission. At the same time, by setting the jitter limit parameter, the problem of unstable transmission caused by frequent link switching can be avoided.

[0128] The data transmission method provided in the embodiment of the present application, for streaming data and batch data with different importance existing in the data middle station, divides the priority of the data according to the importance and data type, and saves it in the corresponding priority queue. For other priority queues except the highest priority queue, based on the improved queue scheduling algorithm, the weight is dynamically determined in combination with factors such as queue priority, delay and load, and the number of polling times of each queue is determined according to the dynamic weight. According to the polling number of each queue, the data is read from the corresponding queue and saved in the virtual queue; the priority queue algorithm is then used to schedule the virtual queue and the highest priority queue to construct an output queue. For the data to be transmitted in the output queue, a suitable target link is selected from the link list for transmission, and through periodic link monitoring, a more suitable link is switched in time according to the changes in the target link. The data transmission method provided by the present application can realize the coordinated transmission of batch data and streaming data, and improve the transmission efficiency and reliability of streaming batch data.

[0129] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0130] It should be noted that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0131] like Figure 4 As shown, the embodiment of the present application also provides a data transmission device, including:

[0132] The acquisition module is used to obtain streaming data and batch data from different data sources;

[0133] A queue construction module, used to divide the streaming data and batch data into multiple priorities according to importance and data type, and store the data of each priority in a corresponding priority queue;

[0134] A first scheduling module, used for scheduling other priority queues except the highest priority queue according to a preset queue scheduling method to generate a virtual queue;

[0135] A second scheduling module is used to schedule the virtual queue and the highest priority queue according to the priority queue algorithm to obtain an output queue;

[0136] A transmission module is used to transmit the data in the output queue.

[0137] For the convenience of description, the above devices are described in terms of functions divided into various modules. Of course, when implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0138] The apparatus of the above-mentioned embodiment is used to implement the corresponding method in the above-mentioned embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0139] Figure 5A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0140] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0141] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0142] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0143] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0144] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0145] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0146] The electronic device of the above embodiment is used to implement the corresponding method in the above embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0147] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0148] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0149] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present disclosure, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0150] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0151] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present disclosure.

Claims

1. A data transmission method, characterized in that: include: Get streaming and batch data from different data sources; According to the importance and data type, the streaming data and batch data are divided into multiple priorities, and the data of each priority is stored in a corresponding priority queue; For queues of other priorities except the highest priority queue, schedule them according to the preset queue scheduling method to generate virtual queues; For the virtual queue and the highest priority queue, scheduling is performed according to the priority queue algorithm to obtain an output queue; The data in the output queue is transmitted.

2. The method according to claim 1, characterized in that According to the importance and data type, the streaming data and batch data are divided into multiple priorities, and the data of each priority is stored in the corresponding priority queue, including: Important data that affects the operation of the power grid is regarded as the highest priority data and stored in the highest priority queue; Streaming data that affects real-time analysis and decision-making of the power grid is treated as high-priority streaming data and saved in a high-priority stream queue; The batch data of power finance is regarded as high-priority batch data and saved in the high-priority batch queue; The power grid feedback streaming data is regarded as medium priority streaming data and saved in the medium priority stream queue; The batch data of power grid design and operation and maintenance are regarded as medium priority batch data and stored in the medium priority batch queue; Treat other data as low priority data and save them in the low priority queue.

3. The method according to claim 1 or 2, characterized in that: For queues of other priorities except the highest priority queue, schedule them according to the preset queue scheduling method to generate virtual queues, including: Determining the dynamic weight of each queue in the other priority queues according to the priority, average delay and load parameters of each queue in the other priority queues; Determining the number of polling times for each queue according to the dynamic weight; Read data from each queue according to the polling times of each queue; The virtual queue is constructed based on the read data.

4. The method according to claim 3, characterized in that Determining the dynamic weight of each queue in the other priority queues according to the priority, average delay and load parameter of each queue in the other priority queues includes: For any queue in each queue, a weight adjustment factor is calculated according to the priority, average delay and load parameters of the queue; The weight of the queue in the next scheduling period is determined according to the weight of the current scheduling period and the weight adjustment factor.

5. The method according to claim 4, characterized in that The weight adjustment factor is calculated based on the priority, average delay, and load parameters of the queue as follows: ΔW i =αP i +βDF i +γQF i (2) Where, ΔW i is the weight adjustment factor of the i-th queue, P i is the priority of the i-th queue, DF i is the average delay of the ith queue, QF i is the load parameter of the ith queue, and α, β, and γ are weight coefficients.

6. The method according to claim 5, characterized in that The weight of the queue in the next scheduling period is determined according to the weight of the current scheduling period and the weight adjustment factor, and the method is as follows: IN i (t+1) =In i (t) +ΔW i (8) Among them, W i (t) is the weight of the i-th queue in the t-th scheduling cycle, W i (t+1) is the weight of the i-th queue in the t+1-th scheduling period.

7. The method according to claim 6, characterized in that According to the dynamic weight, the polling times of each queue are determined by: Among them, N i is the polling times of the i-th queue in the next scheduling cycle, W i is the weight of the i-th queue in the next scheduling cycle, n is the total number of queues of other priority levels, and N is the preset total number of polling times.

8. The method according to claim 1, characterized in that Transmitting the data in the output queue, comprising: Selecting an adapted target link from a pre-constructed link list according to quality requirements of the streaming data and / or batch data to be transmitted in the output queue; The target link is used to transmit the streaming data and / or batch data to be transmitted.

9. The method according to claim 8, characterized in that Transmitting the streaming data and / or batch data to be transmitted using the target link includes: Sending a test data packet for detecting the link status to the target link according to a preset period; Determine whether to switch the target link according to the test result of the test data packet.

10. A data transmission device, characterized in that: include: The acquisition module is used to obtain streaming data and batch data from different data sources; A queue construction module, used to divide the streaming data and batch data into multiple priorities according to importance and data type, and store the data of each priority in a corresponding priority queue; A first scheduling module, used for scheduling other priority queues except the highest priority queue according to a preset queue scheduling method to generate a virtual queue; A second scheduling module is used to schedule the virtual queue and the highest priority queue according to the priority queue algorithm to obtain an output queue; A transmission module is used to transmit the data in the output queue.

Images